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A study  was  made  to  determine  the  feasibility  of  using  videotape  as  the 
presentation  mode  in  a simulation  performance  test  of  certain  tasks  in  the 
Army's  Carpentry  and  Maspnry  MOS.  Two  sets  of  proceduresc,“tT)' task  selec- 
tion procedures,  and  i2)  simulation  procedures',^ were  developed  which  provide 
detailed  guidance  to  future  simulation  test  developers.  A prototype  simula- 
tion test  was  developed  and  validated  against  general  performance  ratings 
and  a similar  written  instrument.  The  fol lowing^ was  concluded:  (1)  applica- 
tion of  the  procedures  enabled  the  selection  of  the  more  appropriate  tasks 
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cability of  audio-visual  simulation  to  test  perceptual  content  was  not 
conclusively  answered,  and  (3)  the  use  of  television  strictly  for  testing 
the  perceptual  content  of  lower  skill  level  motor  tasks  such  as  those 
within  the  Carpentry  and  Masonry  MOS  appears  somewhat  limited;  there 
appears,  however,  to  be  a decided  favorable  attitudinal  bias,  on  the  part 
of  the  test  taker,  towards  television  testing. 
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A companion  volume  was  produced  for  ARI,  ‘Guidelines  for  the  Developers 
of  Videotape  Simulation  Performance  tests, •*  P-78-1,  which  provides  detailed 
guidance  for  test  developers. 
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SUMMARY  AND  CONCLUSIONS 


BACKGROUND 

The  Ai-my  Research  Institute  (ARl)  has,  for  several  years,  been  explor- 
ing various  methods  of  "synthetic  performance  testing"  (Osborn,  1970)  in  an 
effort  to  identify  alternatives  to  the  full  performance  test,  which  is  seen  as 
more  valid,  but  not  as  feasible,  and  the  written  test,  which  is  seen  as  more 
feasible,  but  not  generally  as  valid.  The  development  of  valid  and  reliable 
synthetic  performance  tests  has  significant  impact  on  the  performance- based 
criterion  referenced  Skill  Qualification  Tests  (SQT)  which  form  the  evaluative 
heart  of  the  Army's  Enlisted  Personnel  Management  System.  One  form  of 
synthetic  test  which  appeared  promising  was  audio-visual  simulation  testing 
with  television.  Specifically,  ARl  was  interested  in  assessing  television 
simulation  as  a means  of  presenting  perceptual  and  perceptual  related  psy- 
chomotor tasks  within  the  Army's  Carpentry  and  Masonry  Military  Occupational 
Specialty.  It  is  this  assessment  which  is  addressed  in  this  report. 

PROCEDURES 

The  research  design  entailed  the  completion  of  four  tasks:  (1)  a proto- 
type task  selection  procedure  was  developed  which  ranked  a given  field  of 
tasks  on  the  basis  of  their  common  critical  elements  in  order  to  permit  the 
early  analysis  of  those  tasks  deemed  most  appropriate  for  inclusion  in  a 
simulation  test,  (2)  a simulation  procedure  was  constructed  which  took  tasks 
from  the  task  procedure  and  provided  a structured  method  for  analyzing  the 
common,  critical  elements  in  terms  of  their  perceptual  components  and  their 
feasibility  for  simulation  testing,  (3)  a prototype  simulation  (television)  test 
was  constructed  in  conformity  with  the  task  selection  and  simulation 
procedures,  and  (4)  the  prototype  test  was  evaluated  in  terms  of  its  validity 
and  feasibility  when  compared  with  performance  and  written  tests. 

Achievement  of  a second  contract  objective,  the  construction  of  guide- 
lines for  the  developers  of  similar  tests,  involved  a separate  deliverable  item 
and  is  not  addressed  in  this  report. 


RESULTS 

Both  the  task  selection  and  the  simulation  procedures  provided  task 
element  data  and  guidance  generally  as  expected.  The  prototype  simulation 
test,  when  correlated  with  less  than  ideal  criterion  measures,  was  shown  to 

be  significant  but  not  necessarUy  of  more  validity  than  a similar,  written 
test . 


CONCLUSIONS 

TASK  SELECTION  AND  SIMULATION  PROCEDURES 

The  application  of  the  procedures  enabled  the  selection  of  the  more 
appropriate  tasks  and  task  components  from  a specified  field  of  tasks  critical 
to  MOS  51 A and  51B. 

Use  of  the  simulation  procedures  requires  a greater  expenditure  of 
human  resources  than  may  typically  be  present  in  a test  development  agency. 

APPLICABILITY  OF  A/V  SIMULATION 

The  fundamental  question  of  the  applicability  and  validity  of  A/V  simu- 
lation to  test  perceptual  content  was  not  conclusively  answered  because  of  a 
number  of  problems  discussed  in  the  text  of  the  report. 

The  use  of  television  as  a simulation  means,  strictly  for  testing  the  per- 
ceptual content  of  lower  skill  level  motor  tasks  such  as  those  within  the  car- 
pentry and  masonry  MOS  appears  somewhat  limited;  there  appears,  however, 
to  be  a decided  favorable  attitudinal  bias,  on  the  part  of  the  test  takers, 
towards  television  testing. 
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BACKGROUND 


REQUIREMENT 

Under  contract  DAHC  19-76-C-0032,  Data-Design's  efforts  were  directed 
towards  the  accomplishment  of  two  basic  objectives:  (1)  develop  and  evaluate 
the  methodology  necessary  to  construct  an  audio-visual  simulation  (television) 
performance  test  of  tasks  within  the  Army's  Carpentry  and  Masonry  Military 
Occupational  Specialties  (MOS),  and  (2)  produce  the  guidelines  which  will 
enable  Army  test  developers  to  construct  simulation  performance  tests  for 
similar  MOS. 

Achievement  of  the  first  objective  required  the  performance  of  four 
tasks:  (1)  the  development  of  a prototype  task  procedure  to  analyze  and 

classify  perceptual  and  perceptual  related  psychomotor  tasks  according  to 
their  common,  critical  elements,  (2)  the  development  of  a simulation  procedure 
which  will  take  tasks  from  the  task  model  and  provide  a structured  method 
for  analyzing  the  common,  critical  elements  in  terms  of  their  feasibility  for 
simulation  testing,  (3)  the  development  of  a prototype  simulation  test,  and  (4) 
the  evaluation  of  the  prototype  test,  in  terms  of  its^  validity  and  costs  when 
compared  with  performance  and  written  tests. 

Achievement  of  the  second  objective,  production  of  the  guidelines,  in- 
volved a separate  deliverable  item  under  the  contract  and  will  not  be 
addressed  in  this  report. 

STATEMENT  OF  THE  PROBLEM 


The  Army's  Enlisted  Personnel  Management  System  (EPMS)  is,  to  a great 
extent,  dependent  upon  the  construction  and  utilization  of  Skill  Qualification 
Tests  (SQT)  which  are  fair,  valid,  and  reliable  measurements  of  present 
performance  and  predictors  of  future  performance.  The  procedures  and 
requirements  for  constructing  these  tests  are  specified  by  official  directive 
(ITED,  1976).  To  be  useful,  SQT  must  be  administered  feasibly  in  the  field, 
under  a variety  of  conditions,  to  the  many  thousands  of  soldiers  worldwide. 
SQT  are  performance  based,  criterion  referenced  test  instruments,  which  are 
to  be  constructed  as  hands-on,  full  performance  tests  whenever  they  can  be 
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administered  feasibly  as  such  Feasibility  is  used  here  in  the  sense  of  the 
cost-effective  use  of  limited  resources.  For  many  tasks,  full  performance 
tests  involve  the  dedication  of  costly  equipment  and  manpower  each  time  the 
test  IS  given.  Raters  must  be  trained,  test  sites  and  conditions  must  (to  the 
extent  practicable)  be  standardized,  and  in  many  cases,  key  personnel  are 
assigned  as  raters  and  are  therefore  lost  to  their  primary  duty  stations 
during  test  administration.  When  faced  with  tasks  which  simply  cannot  be 
administered  feasibly  in  the  hands-on  components,  or  if  the  component  is  full, 
the  test  developer  has  two  alternatives:  (1)  the  performance  certification 
component,  or  (2)  the  written  component. 

Evidence  has  shown,  however,  that  paper-and-pencil  tests  are  of  rela- 
tively low  validity  when  correlated  with  performance  tests.  Pickering  and 
Anderson  (1976)  conclude  that  the  correlations  between  performance  and 
theory  tests  and  performance  and  knowledge  tests  "are  not  high  enough  to 
justify  the  substitution  of  job  knowledge  tests  for  job  performance  tests." 
Foley  (1974),  presented  correlations  of  job  performance  tests  with  theory 
tests  and  job  knowledge  tests  and  the  ranges  (.03,  .36)  for  performance  and 

theory,  and  (.10,  .55)  for  performance  and  knowledge,  indicate  a low  validity 

for  these  written  instruments . 

Higher  correlations  between  job  knowledge  and  job  performance  (work 
sample)  tests  were  reported  by  Vineberg  and  Taylor  (1972),  in  their  compar- 
ison of  criterion  instruments  in  four  Army  jobs  (Armor  Crewman,  Repairman, 
Supply  Specialist,  and  Cook).  These  correlations  (with  the  effects  of  time  on 
the  job  partialled  out)  ranged  from  .49  for  the  Armor  Crewman  and  the 
Repairman  to  .65  for  the  Supply  Specialist.  It  should  be  pointed  out,  how- 
ever, that  the  four- jobs  were  categorized  in  terms  of  their  knowledge  and 
skill  requirements  as  shown  in  Table  1.  It  can  readily  be  seen  that  the  high 
correlation  for  the  Supply  Specialist  is  due,  at  least  in  part,  to  the  absence 
of  skill  requirements  for  the  job. 

Indeed,  the  authors  indicate  that  this  relationship  is  of  particular  signif- 
icance for  their  analysis  in  that  the  job  of  the  Supply  Specialist,  "represents 
one  of  the  purest  examples  of  a job  where  knowledge  rather  than  skill  is 
sufficient  to  support  performance"  (p.20). 

Army  test  developers  are  faced  with  essentially  two  unsatisfying  choices; 
(1).  a performance  test  which  is  more  valid  but  typically  not  as  feasible,  or 
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TABLE  1.  FREQUENCY  OF  STEPS  IN  JOB  SAMPLE 
TESTS  CATEGORIZED  ACCORDING  TO  KNOWLEDGE  AND  SKILL  REQUIREMENTS 


Requirements 

Armor 

Crewman 

Repairman 

Supply 

Specialist 

Cook 

Knowledge  Alone 

338 

165 

153 

145 

Cognitive  Skill  and 

0 

4 

3 

11 

Knowledge 

Perceptual  Motor  Skill 

16 

6 

0 

2 

and  Knowledge 

Total  Number  of  Steps 

NOTE:  From  Vinelierg 

354 

and  Taylor  (1972) 

175 

156 

158 

C2).  a written  test  which  is  typically  more  feasible  but  not  as  valid.  Conse- 
quently there  was  a desire  to  develop  a broader  range  of  alternatives.  The 
Army  Research  Institute  (ARl)  began  exploring  various  methods  of  "synthetic 
perfonnance  testing’’  tOsborn,  1970)  in  order  to  develop  alternatives  between 
the  extremes  of  performance  and  written  tests.  One  form  of  synthetic  testing 
which  appeared  promising  was  audio-visual  simulation  as  a means  of  present- 
ing perceptual  or  perceptual  related  psychomotor  tasks. 

ARl  designated  skill  levels  1 and  2 of  MOS  51B  (Carpentry  and 
Masonry  Specialist)  as  an  appropriate  vehicle  for  assessing  the  effectiveness 
of  SQT  testing  via  audio-visual  simulation.  These  skill  levels  are  nonsuper- 
visory . Tasks  such  as  "place  and  finish  concrete,"  and  "construct  and  erect 
wall  fonns"  are  extremely  poor  candidates  for  both  performance  and  written 
tests.  The  expense  of  administering  full  performance  tests  of  these  items  to 
a large,  worldwide  population  would  be  significant  and  likely  prohibitive,  and 
the  validity  of  a written  test  appears  questionable.  Since  both  of  these  tasks 
appeared  to  include  a substantial  perceptual  component  (i.e.  . fine  perceptual 
discriminations  must  precede  motor  behavior),  it  appeared  that  audio-visual 
simulation  was  viewed  as  potentially  an  acceptable  alternative. 

A preliminary  analysis  of  skill  levels  1 and  2 for  the  Carpentry  and 
Masonry  MOS  indicated  that  the  information  was  primarily  equipment  and 
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procedure  oriented.  For  example,  fabrication  of  a frame  is  typically  an 
ongoing  process  which  involves  the  response  to  stimuli  most  frequently  given 
in  the  form  of  visual  or  spatial  cues. 

The  research  team  proposed  that  videotape  would  be  the  most  suitable 
equipment  to  convey  subject  matter  which  is  both  visual  and  spatial  in  na- 
ture. In  comparison  with  the  other  equipments  currently  available  at  each 
active  Army  battalion,  videotape  appeared  to  offer  the  most  advantages  in 
terms  of  production  features  and  message  impact.  Print  messages  have  a low 
sensory  impact.  Essentially,  they  lack  the  capacity  to  attract  busy  individ- 
uals to  their  message.  Audiotape  is  monosensory;  that  is  to  say  it  lacks  the 
visual  element  entirely  and  was  inappropriate.  Slides  or  filmstrips  lack  the 
dimension  of  motion,  and  by  this  deficiency  appeared  to  exclude  any  real 
ability  to  reproduce  the  attributes  of  an  ongoing  procedure.  Of  the  equip- 
ments available  at  the  battalion  sites,  only  the  Beseler  Cue/See  devices  and 
videotape  had  the  ability  to  present  subjects  which  required  motion,  sound  or 
color,  and  there  appeared  to  be  certain  advantages  to  videotape  production 
that  made  a better  choice. 

Working  with  film  can  be  a very  time  consuming  venture.  Once  aU  the 
footage  is  shot,  the  film  is  mailed  away  to  be  developed,  and  when  returned 
it  is  reviewed  and  edited.  If  there  are  any  problems  with  the  film,  or  addi- 
tional scenes  are  needed,  the  whole  cycle  has  to  be  repeated.  In  working 
with  videotape,  there  is  no  time  delay  between  shooting  material  and  review- 
ing it.  Immediately  after  the  footage  is  shot,  it  can  be  reviewed  on-the-spot 
with  a portable  monitor.  Then,  if  additional  shots  are  required,  they  can  be 
taken  at  that  time. 

ARI  concurred  with  the  proposed  test  presentation  mode  and  the  subse- 
quent research  effort  reflected  that  decision. 


LIMITATIONS  ON  PERFORMANCE  OF  THE  RESEARCH 


When  this  research  was  planned,  it  was  anticipated  that  MOS  51 A (Utility 
worker)  and  51B  (Carpenter)  would  be  combined  to  form  a new  MOS  51B 
(Carpenter  and  Mason).  This  combination  would  have  resulted  in  the  con- 
struction of  a Skill  Qualification  Test  (SQT)  for  the  MOS.  The  written  and 
hands-on  components  of  the  SQT  would  then  have  formed  the  evaluative 
criteria  for  the  subsequently  developed  simulation  test.  However,  the  new 
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MOS  was  not  oreatod,  and  as  a oonst>quenot* , no  SQ'l’  was  developed  for  either 
blA  or  hlB.  In  the  absence  of  an  it  was  necessary  to  obtain  criterion 

measures  elsewhere.  The  most  appropriate  alternative  source  available  was 
the  perfonaance  ratings  from  certain  module  tests  jjiven  to  trainees  during 
the  advanced  individual  training  of  both  MOS  51A10  and  51H10.  Thus,  cri- 
teria for  the  validation  of  the  prototype  simulation  test  were  not  item  specific, 
hut  generalized  from  the  module  test  ratings. 

PURPOSE 


The  specific  purpose  of  this  research  was  to  evaluate  a televised  simu- 
lation test  of  a few  tasks  of  an  SQT,  as  a medium  for  testing  perceptual  and 
perceptual  related  components  within  MOS  SIB.  This  research  is  part  of  a 
broader  AKl  effort  to  assess  the  overall  effectiveness  of  an  audio-visual 
simulation  StJT. 

This  evaluation  investigated  both  the  stimulus  and  response  dimensions 
of  these  components  and  the  degree  to  which  they  might  be  faithfully  repli- 
cated by  television.  It  also  included  an  evaluation  of  the  feasibility  of  devel- 
oping and  administering  television  simulation  tests  in  the  field  Feasibility  is 
measured  here  by  two  criteria;  (1)  ability  to  develop  the  tests  using  the 
manpower  typically  resident  in  Amy  test  development  activities,  and  (2) 
administration  costs  which  are  significantly  less  than  those  incurred  in  the 
administration  of  full  performance  tests  of  the  same  tasks. 

DEVELOPMENT  AND  APPLICATION  OF  THE  TASK  SELECTION  PROCEDURES 

Development 


The  Ai*my  test  tleveloper  (today)  can  be  faced  with  a list  of  critical 
tasks  ranging  from  as  few  as  30  to  more  than  250’,  depending  upon  the  MOS. 
Current  guidance  (Osborn,  et  al,  1977)  specifies  that  no  more  than  76  tasks 
may  be  included  in  any  one  SQT . Because  time  is  a resource  typically  in 
short  supply  at  test  development  activities,  an  efficient,  workable  methodology 
for  reducing  the  task  field  to  a manageable  number  is  clearly  desired.  The 
Task  Selection  Procedures  (Appendix  A)  were  designed  to  aid  the  test  devel- 
oper in  the  systematic  reduction  and  ordering  of  a given  field  of  tasks.  The 
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procedures  allow  assossn\cnt  ot'  oach  task,  at  the  task  t'Kaaonl  level,  for  Wt. 
commonality  and  criticality  wh«'n  comparevl  to  llu'  «'ntire  I'ijdd  of  la.sks  in  that 
MOS  at  a j^ivi'n  skill  levt'l.  I'he  result.int  task  list  is  rank  oisK'reil  from  the 
most  common  and  i-ritical  task  to  the  least 

It  was  telt  that  an  t'xistintv  t.'ixonomy  of  ^'t'ni’i'al  vei'hs  mnvht  he  iil«‘nli- 
fied  which  could  b«'  employed  in  dt'V’i'U'pinjj;  llu*  proceilures  ItuiUlin^'  up^'n 
the  pioneenn^^  work  of  I'otlerman  (Itlh;)).  Merliner.  el  al  tUHvl)  ilassifieii 
tasks  into  ti>ur  maji'r  cal<'j;x>ries . i iv.  pei‘c«‘ptual . mediational. 
communicational.  and  motor  processes  Ut'nnetl  (.1071’)  fai'lor  analy.’tal  stu- 
dent judtO"t'*'l ^ '■'I  task-related  verbs  and  found  four  c.ate^jories  I'f  task  vari- 
ables which  h«‘  labeled  as  cognitive,  social,  procislural . ami  physical  Other 
task  taxonomies  were  developed  by  Hague  11%:;).  Kolley  (llHvl).  ami  Fleish- 
man and  his  associates  (summarized  in  l97.S>  among  others. 

Is  it  possible  to  construct  a univers.il  task  classifii-ation  moili'l’  Fh'ish- 
man  (lilTh)  states:  "The  search  for  a single  gent'ral  taxonomy  is  not  likely  to 
be  successful  for  all  purposes.  We  may.  indeed,  neeil  several  task  classifica- 
tion systems  for  several  purposes,  with  the  linkage  between  them  understooil 
and  specified  . . Taxonomies  are  not  out  there  to  he  discovereil.  siune 
invention  is  required."  (p.  l\‘17V 

Discussions  with  senior  personnel  at  the  SQT  branch.  U S Aimy  F.ngi- 
neer  School,  Fort  belvoir,  VA,  and  at  Kngineer  training  bidgades  at  Fort 
beonard  Wood.  MO.  led  to  the  belief  that  any  single  list  of  verbs  ran  the 
risk  of  being  too  general.  This  became  especially  evident  upon  studying  tht' 
task  descriptions  of  MOS  hi  A.  Specialized  verbs  such  as  screed,  Iixuvel.  and 
float  are  unique  or  have  definitions  which  are  unuiue  to  tasks  dealing  with 
concrete.  Obviously,  the  difference  between  "floating"  conenue  (part  of  the 
finishing  process)  and  floating  a rivtu'  (as  in  tactu'al  amphibious  opi'ration)  is 
significant  and  to  class  them  as  common  behaviors  on  the  basis  of  their  cimu- 
mon  verbs  would  be  misleading 

Procedures  were  developed,  then,  to  allow  individual  test  deveK'pment 
organizations  to  utilize  simple  l.ixonomii's  in  which  the  verbs  are  s[>ev'ific  to 
each  MOS  and  skill  level  Tin'  user  is  ilirected  to  i-onstrin'l  a matrix  in 
which  critical  tasks  are  listed  horr/.ontally  ami  the  key  verbs  are  listed  verti- 
cally. The  matrix  allows  for  a heavier  weighting  of  viu'bs  or  tasks  whiidi  are 
considered  by  subject  matter  experts  to  be  highly  ciitical  The  v«>ibs  listed 
in  the  vertical  column  can  be  taken  from  the  perbumance  steps  as  listi'd  in 
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the  Task  Data  Cards,  Job  Task  Summary  Sheets,  or  whatever  job/ task  anal- 
ysis form  is  in  use  at  the  particular  organization 

The  output  of  these  procedures  is  a rank  ordering  of  tasks  in  tenns  of 
criticality  and  commonality  at  the  task  element  level 

Application 

An  inventory  of  tasks,  designated  as  critical  by  the  Army  Engineer 
School's  Task  Analysis  Branch,  was  obtained  and  plotted  in  the  selection 
matrix  as  shown  in  Figure  1.  Figure  2.  the  Job  Task  Summary  Sheet,  is 
typical  of  the  task  statements  used  as  input  data. 

In  order  to  demonstrate  the  reliability  of  the  selection  procedures,  the 
same  Ust  of  critical  tasks  was  given  to  a senior  noncommissioned  officer  in  the 
test  development  activity  at  the  Army  Engineer  Schotd,  Fort  Belvoir,  VA 
The  NCO  applied  the  procedures  to  the  list  and  with  one  exception,  ranked 
the  tasks  in  the  same  order  as  the  first  application  The  difference  was 
caused  by  the  NCO's  plotting  the  behavior  designator  "maintain  tools"  as  the 
most  common  in  the  entire  list . He  reasoned  properly , that  the  behavior  of 
maintaining  tools  was  an  implied  step  in  the  performance  of  every  task  in  the 
list,  and  plotted  it  as  such.  This  points  up  a major  limitation  to  the  matrix; 
i.e.,  its  dependence  upon  detailed  task  statements  as  input  data.  Strictly 
interpreted,  the  procedures  allow  the  user  to  ignore  any  behaviors  not  expli- 
cit in  the  task  statements. 

Another  NlX)  was  asked  to  use  the  procedures  to  order  a list  of  tasks 
from  the  MOS  in  which  he  was  a subject  matter  expert;  i e.  . MOS  62F  (Crane 
Operator).  The  procedures  are  generalizable  to  the  extent  that  he  was  able 
to  complete  the  matrix  ami  rank  order  a sample  list  of  20  tasks,  using  JTSSs 
as  input . 

DEVELOPMENT  AND  APPLICATION  OF  THE  SIMULATION  PROCEDURES 

Development 


These  procedures  provide  the  test  develojier  with  a means  of  si-lei  ting 


ajipropriati' 


jierceptual  content  foi-  A/V  simulation  tt'sts  and  for  develojung 
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valid  criterion  referenced  simulation  tests.  The  followinR  features  arc  pro- 
vided . 

(1)  An  output  of  a simulation  test  that  can  be  presented  in  a standard 
situation  to  examinees  at  many  sites. 

(2)  A procedure  for  selecting  only  critical  components  of  tasks. 

(3)  A procedure  for  presenting  items  in  the  job  functional  context. 

(4*)  A procedure  for  sequencing  activities  as  they  will  occur  on  the  job. 

(5)  A procedure  for  identifying  relevant  stimuli  for  decision  making. 

(6)  A procedure  for  identifying  appropriate  test  responses  to  test 
stimuli. 

(7)  Procedures  which  can  be  applied  to  A/V  simulation  test  construction 
with  task  data  from  other  MOSs. 

Development  of  the  procedures  was  largely  concerned  with  the  (juestion 

of  how  to  construct  a criterion-referenced  test;  a subject  that  has  received 

considerable  attention  in  recent  years  (Laabs,  Main,  Abrams,  Steinnemann, 
1975;  Osborn,  1973;  Panitz  and  Olivo,  1970;  Swezey  and  Pearlstein,  1974, 
Osborn,  Campbell,  Ford,  Hirshfeld  and  Maier,  1977).  Two  major  differences 
between  the  simulation  procedures  and  those  strictly  concerned  with  job 
sample,  criterion-referenced  tests  are  that:  (1)  these  procedures  are  also 
concerned  with  assuring  appropriate  simulation  of  test  items  and  (2)  the 
procedures  assume  that  each  task  element  is  associated  with  a behavioral 
objective  which  specifies  that  the  examinee  should  be  able  to  perform  the  task 
element  under  job  conditions  at  a mastery  level.  (This  is  stated  as  an  as- 
sumption because  both  sets  of  procedures  start  with  the  task  element  as  a 

given  and  can  only  assume  that  it  was  property  derived  in  a task  analysis 

model. ) 

The  simulation  procedures  are  presented  in  the  format  of  an  algorithm 
which  leads  the  test  developer  through  the  following  sequence  of  events: 

(1)  Analysis  of  the  selected  critical  task  elements  to  identify  percep- 
tual, cognitive,  and  motor  components. 

(2)  Selection  of  the  perceptual  components  for  videotaped  simulation  and 
assignment  of  motor  and  cognitive  components  to  performance  and 
written  tests. 

' An  informal  review  of  the  simulation  procedures  by  test  »levelopmeiit 
personnel  at  the  Army  Engineer  School  resulted  in  the  terms  cognitive  and  motor 
being  redesignated  as  decision  and  action.  This  was  done  to  enhance  user 
acceptability  and  is  reflected  in  subsequent  element  analysis  tables. 
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Aaemble 

Bore 

Mamuin  (TooisI 
Cut  (CorKrtte) 

Cut  (Wood) 

ConsolKtote  (V)b.  Cone) 
Check  (Inspect) 

Edge  (Concrete) 

Entpieoe 
Excevate 
Finish  (Concrete) 

Identity 

Interpret  (Dvwgi) 

Lemirute 

Ley  (Brick.  Block) 

Uy  (Sheathing,  Shingles) 
Level 

Lower  (w/iecks) 

Meeaure 

Mix 

Neil 

Oil/Wet  (Forms  & Bolts) 
Operate  (Chain  Sow) 
Place  (Cortcrete) 
tosition 
Plumb 

Raise  (w/jecks) 

Sharpen  ITooH) 

Shape  (Pites) 

Square 

Sharpen  (Posts) 
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CRITICAL  TASKS 


! 

1 

i 

• 

« 

a 

E 

• 

c 

3 

S 

3 

1 

• 

O 

J 

s 

2 

1 

• 

C 

Construct  building  roofs. 

11 

o • 

is 

is 

w u 

*f  s 
isi 

III 

III 

Install  arschor  bolts  in  concrete 

Construct  lOints  in  concrete 

Prepare  timber  piles  for  driving 

Assist  m the  construction  of  pile  bents 
for  bridges,  piers  and  wharves  ' 

Place  and  finish  concrete. 
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TASKS.  RANK  ORDERED 


1 Construct  and  reftlace  footers  ar>d  columns. 

3 Frame  walls  ar>d  partit«or)s. 

3 Assemble  roof  trusses  using  template 
Construct  building  roofs 

4 Fabricate  and  imtall  girders,  floor  loists 
and  bridging 

Install  roof  trusses. 

Assist  in  the  fabrication  artd  irntallation  of 
forms  for  cortcrete  footers,  fourtdations.  etc. 
Imtall  anchor  bolts  m concrete. 

Prepare  timber  piles  for  driving. 

Auisi  in  the  fabrication  ar>d  irntallation  of 
forms  for  walls,  columm,  stairs  ar>d  floor 
slabs. 

5.  Cut  and  install  batter  boards. 

Imtall  artd/or  replace  subfloor. 

6.  Place  and  finish  corKrete. 

7.  Comtruct  (Oints  in  corscrete. 

6.  Maintain  carpenter  tools  and  equipment. 

9.  Identify  construction  material  by  type  ar>d  si/e 

10.  Assist  in  the  construction  of  pile  bents  for 
bridges,  piers,  and  wharves. 
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Figure  1.  Task  Selection  Matrix 


TASK:  Pldce  arid  ftmsh  concrt^ie  TASK  CRITICALITY 

{CIRCLE  ONE)  C I N 

CONDITION(S):  Given  insijlled  foinis,  ntiKOd  concrete,  tidnsfXjrlmg  equipment 
jnd  masonry  kit.  an  fPMS  51B10  js  a conciele  const  crew  member  under 
close  supervjsron  of  EPMS  51B20  arxl/or  51H30 

STANDARD{S):  lAW  const  prints,  directions  of  supervisor  and  TM.  concrete 
will  be  placed  properly  into  forms  and  finished  to  prescribed  finish. 

REFERENCES:  TM  5-742  Concrete  and  Masonry  June  70 


ITEP  TASK  REFERENCE 

ARTEP  TASK  REFERENCE.  

APPLICABLE  MOS  — 

SKILL  LEVEL ! 

%PERFORMING  TASK  --  

NO.  PERFORMING  TASK 

FREQUENCYOFPERFORMANCE  _ 

JOS  ANALYST 

DATE  PREPARED  ^ Ap„i  75 


STEPS  IN 
PERFORMANCE 


1 Place  plastic  concrete  into 
forms  for  slab. 


2.  Place  concrete  into  wall 
tseam,  and  girder  forms. 


3.  Consolidate  concrete  using 
mechanical  vibrator. 


4.  Screed  concrete  to  finish 
grade. 


5.  Assist  finishing  concrete 
using  wood  float. 


6.  Assist  finishing  concrete 
using  steel  trowel. 


7.  Assist  finishing  concrete 
using  broom. 

8.  Edge  concrete. 


STANDARD  OF 
PERFORMANCE 

Placemen!  of  concrete  will  start  at  far  end  of 
slab  and  each  batch  will  be  dum|)ed  against 
previously  placed  batch  as  directed  by  crew 
chief. 

Concrete  will  be  placed  m 6"  to  24"  lifts 
only,  and  maximum  free  fall  of  concrete 
will  be  5'  when  placing  concrete  into  wall 
forms.  Concrete  will  be  placed  from  each 
end  and  work  to  center  of  form  when 
placing  in  beam  or  girder  forms  under 
close  direction  of  crew  chief. 

Vibrator  will  be  inserted  into  concrete  at 
approximately  18"  intervals  for  5 to  15 
seconds.  Vibrator  will  be  lowered  using  its 
own  weight  to  penetrate  through  several 
inches.  Vibrator  will  be  used  under  close 
supervision  of  crew  chief. 

Screed  board  will  be  placed  fiat  on  wood  or 
steel  forms  moving  across  placed  concrete 
with  a sawing  action  and  forward  motion. 
Screeding  will  be  done  twice  over  area  to 
remove  excess  concrete  brought  up  by 
first  screeding. 


^MATERIALS, 

TOOLS, 

EQUIPMENT 

Installed  forms, 
mixed  concrete, 
wheel  barrow  or 
loading  equipment, 
shovels,  rakes. 


(3)  Ki<’ntifii'atK>n  i>f  p«>ii'i'plnal  I'onipoiunt  ^ in  vt'iy  spci'il’n'  siimulw. 
U'rms  to  onablo  a diMi'iimnation  of  whothiM'  tho  A \’  luoiii.i  o.ni 
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prosont  tho  .stimulus  with  ailt'nuato  fuli-lity 
t,  l>  Klontifioation  aiul  anal\..i..  of  tin-  job  ami  toi.t  takin>i  rc'spon^o  .ill 
of  tho  stimuli  to  bo  smut  la  toil 

t,!0  lilontifioation  of  rt'lovant  stimuli  in  tho  job  oiu'ii'onmont  to  insiiro 
tho  optimization  of  oontoxtual  ouos  m tho  simulation  tost 
i,h)  Assossmont  of  tho  ailoquaoy  of  tho  simulation  iiuulo  in  j'rosont iiij;'  all 
of  tho  stimuli  to  bo  simulatoil 
('!)  Soiiuonoinj;'  tost  stimuli  into  a tost  format 
t,S>  SoriptiiiK'  tho  A/V  tost 

Subjoot  mattor  oxport  roviows  of  tin'  tost  with  n'visioiis  as  in'oos- 
sary 

tlO^  Dotormination  of  tost  validity 

Tho  alfvorithm  foniiat  was  solootoil  as  most  appix'priati'  fv'r  j;'uulinj;'  .Army 
tost  ilovolopmont  porsonnol  throuj;h  a oouiplox  pivvoss  It  is  supploiiiontoil  by 
narrativo  and  oporational  oxamplos  of  oaoh  stop  to  onhanoo  usor  aoroptalnlity 
It  also  inoorporatos  tin'  uso  of  I'urrt'iit  Army  ti'st  do\'t'lopmont  dooumontat ii'ii 

Hooo)jui7.m^  that  tho  intoiniod  usor  was  unhkoly  to  bo  an  oxjn'it  in 
vidiv-pivdiii'tion . tho  jiriv't'diiri's  rail  foi'  oarly  partioij'.ition  of  a niodia 
oxport  in  tho  dovolopnn'nt  pivooss,  and  also  roforonoo  an  apjn'iidi'd  nontooh- 
nioal  disoussion  of  tho  vidiM-produotion  pivooss. 

Application 


Tho  tasks  idontifiod  as  hi>fhly  oomiiion  and  oritioal  woro  thon  analyzt'd 
through  tho  task  soUvtion  pix'i'oduros  As  a rosult . throo  tasks,  yp  main- 


tain oarpi'iitor  tiX'ls  and  I'liuipmont . fill 
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forms, and  (.d)  plaoo  and  finish  oonoroto 
simulation  tost 


I'onstriu't  and  oroot  oonort'tt'  wall 
wi'ft'  soh'oti'd  for  inolusion  in  tin' 


Pining  tho  tiist  subjoot  in.attoi  oxpoit  roviow  ot  tho  oaiutul.ito  t.^sks, 
porsonnol  .it  tho  P.S.  Army  Tr.i  iiiiiig  I'oiit  oi  iKiigiiiooil,  Koit  l.ooiiaul  Wood , Mo  , 
pointoil  out  th.it  tho  t.isk  "ronstriiot  .nut  iopl.no  toi'tois  .nut  loliuiiiis w.is 
raroly  praotiood  anymoio.  Vho  t.isk  w.is  diso.iidod  aiul  lopl.iood  with  oiu"  ijiiitt' 
similar;  i.o.,  "ronstiiiot  and  oroot  torms  tor  ooiu' i ot  *'  wa  I I s , " .Subjoot  iii.ittoi 
exports  .It  tho  Kiiginoor  Soluiol  woro  ooiisiiltod  .nut  agiood  to  this  ohaiigo  I'tie' 
rom.iining  disoussion  will  t raok  tho  task  displavod  in  Kigino  O’l.no  and  Kiiiish 
I'onoroto)  through  tho  koy  stops  in  t tio  simnlatnMi  tost  dovo  U'piiit'iit  j'loooss 
I’ablo  1 shows  tho  task  olomont  analysis  in  toims  ot  its  poi  lOjU  ua  I , oognitivo 
aiidmotoi  ooraponont  s , aiul  displays  tho  lolatod  stimulus  vaiiablos 
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TABLE  2.  ELEMENT  ANALYSIS  TO 
IDENTIFY  COMPONENTS  AND  STIMULUS  VARIABLES 

TASK:  PLACE  AND  FINISH  CONCRETE  (SLAB) 


Critical  Elements 

1.  Place  plastic  concrete 
into  forms  for  slab 

2.  Screed  concrete  to 
finish  grade 


3.  Assist  finishing 

concrete  using  wood 
float 


Critical 

Components 

Action 


Perception/Action 


Perception/Action 


Assist  finishing 
concrete  using 
steel  trowel 


Perception/Action 


Stimulus  Variables 


Uniform  level. 
Absence  of  gross 
low  or  high  spots. 

Firmness  of 
concrete  when 
pressure  is 
applied.  How  far 
foot  sinks  in. 
Amount  of  hydra- 
tion which  has 
taken  place. 
Presence  or 
absence  of  low 
or  high  spots  as 
indicated  by  water 
pockets. 

Presence  or 
absence  of  aggre- 
gate on  the 
surface. 

Firmness  of 
concrete. 

Presence  or 
absence  of  water 
sheen  on  surface 
of  concrete. 

Sound  of  trowel 
when  it  strikes 
surface  of 
concrete. 


The  stimulus  varuihles  of  erifieal  eloments  jvuijjed  U'  have  peiH'eplual 
eomponeitts  were  then  analyzed  in  terms  ot  tht'ii"  impi>rlanee  to  task 
perfonnanee  This  process  is  depicted  in  Table  a Table  I depicts  the  pre- 
lutiinary  test  mode  selection'-^  and  stimulus/ response  similarity 

Finally.  Table  5 depicts  preliminary  judumu'iits  as  to  possible  respons<-s 
on  the  TV  test 

The  test  items  were  developed  in  coordination  with  personm'l  ;it  Forts 
Helvoir  and  l.tvnard  W(.\hI.  based  on  the  element  analysis  ilata  generateil  in 
the  simulation  procedures  'The  items  were  thtui  toniiatted  to  a teli'vision 
script  t8ee  Appendix  and  reviewed  by  subject  matter  experts  Items  from 
Unit  d (.Placing  and  Finishing  t'oncrete)  of  the  test  art'  shown  in  I able  h to 
illustrate  the  transference  of  critical  task  element  and  stimulus  variables  to 
test  items.  Certain  other  items;  i.e  , 17  and  18  of  Unit  8 (Answt'r  Sht't't . 

Appendix  C),  were  added  at  the  request  of  Fort  Leonard  Wood  personnel  to 
enhance  the  task  continuity  and  job  context  realism  of  the  score;ible  unit  ;ind 
the  test  as  a whole. 

SCOPE  AND  LIMITATIONS  OF  THE  SIMULATION  PROCEDURES 

Kxperience  gained  during  tht*  use  of  tlu'  proceilun's  and  the  pc'rtormanci' 
of  the  research  has  enabled  the  identification  of  a number  of  factors  which 
may  limit  their  effective  use. 

Availability  of  Detailed  Task  Analysis 


The  procedure  assumes  as  its  input . task  data  in  whii'h  coiiipvMient  skills 
and  knowledges  have  been  identifieil  'The  pivcevlure  cannot  be  used  etti'c- 
tively  if  only  general  task  statements  are  available 


^ As  mentioiieil  earlier,  it  was  cent  r.^ctl).^l  ly  ileteiniiiied  th.rt  vuleot.ipe 
Itelevisioii)  would  be  the  presentation  mode  tor  the  prototype  test  Television 
seemed,  to  the  research  team  and  to  ARl  , to  be  the  most  .ippropi  i at e medium  tor 
simulating  the  perceptual  content  ot  the  tasks  within  MOS  SIR.  Ihis  vlecision  was 
based  on  a review  and  preliminary  analysis  ot  the  original  task  list  and  ot  the 
nature  of  the  overall  research  ettort 
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TABLE  3.  ELEMENT  ANALYSIS  TO  DETERMINE  THE  IMPORTANCE  OF 
STIMULUS  VARIABLES  ASSOCIATED  WITH  A CRITICAL  ELEMENT 

TASK:  PLACE  AND  FINISH  CONCRETE 


Stimulus  Variables 

Unique 

Essential 

Importance 

Critical  Element:  Screed  Concrete 

to  Finisti  Grade 

a 

Uniform  level  of  concrete 
(level  with  form) 

Yes 

Yes 

Very 

tv 

Absence  or  presence 
of  gross  high  or 
low  spots 

Yes 

Yes 

Very 

Critical  Element  Assist  Fimshinij 

Concrete  Using  Wood 

Fkxit. 

a. 

Firmness  of  concrete 
when  pressure  is 
applied  (How  far 
foot  sinks  in) 

No 

Yes 

Moderately 

tv 

PrestMice  or  absence 
of  low  high  spots  as 
indicated  by  water 
pockets 

Yes 

Yes 

Veiy 

c. 

Presence  or  absence 
of  aggregate  on 
surface 

Yes 

Yes 

Very 

Critical  Element:  Assist  in  Finishing  Concrete  Using  Steel  Trowel. 

Firmness  of  concrete 

No 

Yes 

Moderate 

Presence  or  absent’ 
of  water  sheen  on 
concrete 

Yes 

No 

Moderate 

Sound  of  trowel 
when  It  strikes 
surface 

No 

No 

Mcxferate 

TABLE  4.  ELEMENT  ANALYSIS  FOR  PRELIMINARY 
DETERMINATION  OF  TEST  MODE 

TASK;  PLACE  AND  FINISH  CONCRETE 


Stimulus  Variables 

Importance 

Realism 

Response 

Similarity 

Recommended 

Format 

Critical  Element:  Screed  Concrete 

to  Finish  Grade. 

a. 

Uniform  level  of  concrete 
(level  with  form) 

Very 

TV/Still 

Adequate 

TV 

b. 

Absence  or  presence  of 
gross  high  or  low  spots 

Very 

TV 

Adequate 

TV 

Critical  Element:  Assist  Finishing  Concrete  Using  Wood  Float. 

a. 

Firmness  of  concrete  when 
pressure  is  applied 

Moderate 

TV/Still 

Adequate 

TV 

b. 

Presence  or  absence  low/ 
high  spots  as  indicated 
by  water  pockets 

Very 

TV/Still 

Adequate 

TV 

c. 

Presence  or  absence  aggre 
gate  on  surface 

Very 

TV/Still 

Adequate 

TV 

Critical  Element:  Assist  Finishing  Concrete  Using  a 

Steel  Trowel 

a. 

Sound  of  trowel  striking 
surface  of  concrete 

Moderate 

TV/Slide 
& Sound 

Adequate 

TV 

b. 

Presence/absence  water 
sheen  on  concrete 

Moderate 

TV/Still 

Adequate 

TV 

c. 

Firmness  of  concrete 

Moderate 

TV 

Adequate 

TV 

I 

i 
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TABLE  5.  ELEMENT  ANALYSIS  TO  DETERMINE 
POSSIBLE  TEST  RESPONSE 

TASK;  PLACE  AND  FINISH  CONCRETE 


Stimulus  Variable 

Job  Response  Possible  Responses  on  TV  Test 

Screed  to  Finish  Grade. 

1. 

Concrete  level  with  form 

1. 

Show  comparator  slabs 
or  simply  show  one  slab 
at  a time  and  examinee 

a.  Above  form 

a. 

Remove  excess 

makes  go/ no  go  decision 

b.  Below  form 

b. 

Add  concrete 

on  each  slab. 

2. 

Absence  of  gross  high  or 

2. 

Show  comparator  slabs 

low  spots 

or  simply  show  one  slab 
at  a time  and  examinee 

a.  Absence 

a. 

None 

makes  go'no-go  decision 

b.  Presence  high  or  low 

b. 

Add  or  remove  concrete 

on  each  slab. 

Assist  Finishing  Concrete  Using  Wood  Float. 

1. 

Firmness  of  concrete  when 

1. 

Examples  of  concrete  in 

pressure  is  applied 

varying  stages  of  setting. 

a.  If  ok 

a. 

Begin  screeding 

b.  If  too  wet 

b. 

Allow  more  time  for 
concrete  to  set 

2. 

Presence/absence  of  high  or 

a. 

If  present,  float  to  remove 

low  spots  as  indicated  by 

add  or  remove  concrete 

water  pockets 

if  necessary 

3. 

Presence  or  absence  of 

a. 

If  present,  float  to 

Visual  cue  go.mo  go 

aggregate  on  surface 

b. 

remove 

No  action  if  absent 

Assist  Finishing  Concrete  Using  Steel  Trowel. 


a. 

Firmness  of  concrete 
(pressure) 

a.  If  firm,  begin;  if 
not,  wait 

Show 

judge 

hand  on  concrete  to 
firmness. 

b. 

Presence/absence  of  water 
sheen  on  surface 

c. 

Sound  of  trowel  striking 
surface 

c.  If  ringing  sound, 

begin;  if  dull  scrape. 

Show 

demonstrator. 

wait 
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TABLE  6.  RELATION  OF  SIMULATION  TEST 
ITEMS  TO  CRITICAL  ELEMENTS  AND  STIMULUS  VARIABLES 

TASK:  PLACE  AND  FINISH  CONCRETE 


Performance  Step/ 

Critical  Element  Stimulus  Variable  Simulated  Test  Item 


Assist  finishing  concrete  a.  Firmness  of  concrete 
using  wood  float  when  pressure  is 

applied 


Item  25.  Is  this  concrete  ready 
to  be  floated? 


b.  Amount  of  hydration  (VISUAL.  PAN  OF  SURFACE, 
which  has  taken  place  showing  adequate  hydration. 

HOLD  on  demonstrator  putting 
his  foot  into  surface  to  check 
firmness.) 


A.  Yes 

B.  No 

C.  The  information  is 
insufficient. 


c.  Presence  or  absence 
of  low  or  high  spots 
as  indicated  by  water 
pockets,  ridges 

d.  Presence  or  absence 
of  aggregate  on 
surface 


Item  27.  Which  slab  has  been 
properly  floated? 

(VISUAL:  PAN  OF  SURFACE, 
of  four  comparator  slabs,  three 
show  water  spots,  ridges  or 
aggregate.  One  is  proper.) 

A. 

B. 

C. 

D. 


Assist  finishing  concrete  a.  Firmness  of  concrete 
using  steel  trowel 


Item  28.  Look  at  this  concrete. 
Is  it  ready  for  first  troweling? 


b.  Presence  or  absence 
of  water  sheen  on 
surface 

c.  Ringing  sound  of 
trowel  when  it  strikes 
surface 


(VISUAL;  PAN  OF  SURFACE, 
to  show  slight  water  sheen. 
HOLD  on  demonstrator  as  he 
applies  trowel.)  AUDIO.  Sound 
of  trowel  ringing  on  surface. 

A.  Yes 

B.  No 

C.  The  information  is 
insufficient. 
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Availabitity  of  Appropriate  Perceptual  Content 


It  is  obvious  that  most  human  behavior  occurs  within  the  context  of  the 
perceived  environment;  hence  most  tasks  are  said  to  have  a perceptual 
component.  The  simulation  procedure,  however,  is  designed  to  identify  sit- 
uations in  which  the  perceptual  component  is  the  critical  component  of  appro- 
priate task  oriented  behavior.  Such  situations  are  in  fact  relatively  rare. 
Typically,  people  learn  to  recognize  job  related  stimuli  and  errors  occur  as 
a consequence  of  inaccurate  decisions  or  motor  responses.  Baldwin  (1971) 
discusses  industrial  training  and  notes  the  importance  of  perceptual  learning. 
For  example,  the  TV  repairman  must  learn  to  identify  hums  at  60  Hz.  120  Hz, 
and  15,750  Hz,  and  he  must  learn  to  identify  odors  such  as  burned  insulation 
or  a burned  out  transformer.  Baldwin  points  out,  however,  that  these  stimuli 
are  sufficiently  discrete  that  testing  a graduate  is  unnecessary . 

An  example  of  an  area  in  which  perceptual  judgment  continues  to  be 
a major  factor  in  performance  after  training  is  the  auto  mechanic's 
task  of  using  feeler  gauges  to  adjust  valves  or  points.  Here  proprioceptive 
sensitivity  remains  the  key  element  to  accurate  performance. 

In  the  present  application  of  the  procedures,  it  was  difficult  to  select 
appropriate  test  content  because  tasks  associated  with  levels  one  and  two  of 
the  51B  MOS  have  few  critical  perceptual  requirements.  For  evr,.nple.  the 
task  of  jointing  and  sharpening  a saw  is  described  (in  the  technical  manual) 
as  having  considerable  perceptual  content,  but  in  practice,  carpenters  rely 
on  learned  motor  responses  to  assure  that  they  have  properly  performed  the 
task.  Also,  the  determination  of  a level  surface  would  be  a perceptual  prob- 
lem if  no  aids  were  available,  but  it  becomes  a cognitive  problem  when  the 
task  is  to  decide  whether'  the  level  is  sufficient  when  the  carpenter  recognizes 
that  the  edge  of  the  bubble  in  the  carpenters  level  is  slightly  over  the  edge 
of  the  line. 

Experience  of  the  User  of  the  Procedures 

The  algorithm  was  intended  for  use  by  a relatively  unsophisticated  test 
developer.  Experience  now  suggests  that  some  concepts  and  procedures 
involved  are  relatively  strange  to  this  target  individual  and  that  more  exper- 
ience or  training  may  be  a prerequisite  for  proper  use. 
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Availability  of  Manpower  to  Support  User 


The  procedure  involves  the  frequent  use  of  from  2 to  5 subject  matter 
experts , as  well  as  the  requirement  for  a media  expert  and  for  examinees 
during  the  assessment  of  test  validity  and  reliability.  The  research  te.um 
experienced  difficulty  in  obtaining  part  of  the  support  specified,  and  this  is 
not  attributed  to  a lack  of  cooperation  with  the  team  in  particular.  It  is 
noted  that  few  SQTs  are  fully  validated  in  complete  accordance  with  present 
doctrine.  This  is  probably  the  most  limiting  factor  with  respect  to  the  use  of 
the  procedures  in  the  Army  test  development  environment;  in  fact,  the  lack 
of  manpower  support  was  the  principal  reason  that  the  procedures  were  not 
completely  followed  in  this  research. 

PRODUCTION  AND  EVALUATION  OF  THE  SIMULATION  TEST 


Production  of  the  simulation  test  took  place  at  Fort  Leonard  Wood,  MO, 
where  both  MOS  51 A and  51B  are  taught.  Instructors  from  the  courses  were 
used  as  "actors"  in  the  test  since  they  were  already  experts  in  the  tasks  and 
it  was  fell  they  could  more  easily  adjust  their  work  to  suit  the  special  re- 
quirements of  recording  the  effort  on  videotape. 

PRODUCTION  OF  THE  TEST 

The  simulation  test  consisted  of  three  parts,  each  representing  a score- 
able  unit.  These  units  were: 

• Unit  1:  Hand  Tool  Maintenance  and  Material  Preparation  (11  items) 

• Unit  2:  Erecting  Wall  Forms  (9  items) 

• Unit  3:  Placing  and  Finishing  Concrete  (7  items) 

The  first  and  third  units  consisted  of  four  alternative  multiple-choice 
items;  the  second,  two  alternatives  ("correct"  and  "incorrect").  In  addition, 
the  second  unit  contained  a number  of  unalerted  safety  violations,  and  exam- 
inees were  asked  to  identify  the  type  of  safety  violations  by  a code  number 
as  follows ; 

1 . Failure  to  ground  electric  tcwls  or  equipment  properly . 

2.  Failure  to  wear  protective  gear  when  necessary. 

3.  Use  of  tool  in  a hazardous  manner. 

4.  Unsafe  vehicle  operating  procedures. 
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Tho  lojjistii's  of  produoinj;'  tho  tost  nuuio  a nusinin^ful  rovi«“w  oyolo 
impossiblo  prior  to  validation  of  tho  tost.  For  oxamplo,  tho  training  sohoilulo 
at  Fort  I,«‘onar<l  Wcwd  was  such  that  tho  intoinlod  samplo  population  would  ho 
available  for  only  one  day  and  on  a date  whioh  ooviUl  not  be  adjusted  without 
a seven- week  delay.  Therefore,  it  was  necessary  to  carry  out  the  test 
validation  on  the  Tuesday  following  the  Friday  completion  of  the  draft  test. 

However,  the  day  before  the  tost  was  administered,  it  was  shown  to  a 
group  of  seven  instructors  from  tho  carpentry  course,  as  part  of  the  vali- 
dation process.  There  was  agreement  with  the  scoring  key  (.coi'rect  and 
incorrect  classification  of  alternatives). 

IDENTIFICATION  OF  EXTERNAL  CRITERIA 

As  noted  in  the  background  section  of  this  report , Scj  T items  were  not 
available  as  external  criteria  for  purposes  of  validation.  Thus  it  was  not 
possible  to  validate  performance  on  each  critical  task  element  in  the  simulation 
test  against  peiTormance  of  the  identical  critical  task  element  in  a performance 
test.  'The  only  available  performance  related  data  for  pui'pose  of  validation 
were  ratings  which  wei’e  associateti  with  each  man's  perfonuanc<*  in  Carpentry 
and  Utility  Worker  'Training.  'The  module  scores;  (1)  were  based  on  more 
general  observations  than  would  be  required  for  assessment  of  performance  at 
the  level  of  the  critical  task  element,  (2)  included  observations  of  task  ele- 
ments which  were  perhaps  related  to,  but  not  part  of  the  simulation  test,  and 
(3)  did  not  include  the  observation  of  all  task  elements  which  were  included 
in  the  simulation  test . 'The  soldiers  included  in  this  sample  had  been  ob- 
served in  the  perfoniiance  of  the  course  modules  from  two  to  seven  we«'ks 
prior  to  their  performance  of  the  simulation  test;  all  were  trainees  who  hail 
received  further  training  during  this  period. 

For  each  unit  of  the  test,  the  one  or  more  module  ratings  that  most 
closely  matched  the  content  of  the  test  was  used.  Thus  for  Unit  I of  the 
simulation  test,  which  consisted  largely  of  carpentry  related  tasks,  the 
"Building  (.’onst ruction"  module  ratings  were  useil  for  Carpenter  trainees  and 
the  "Carpentry"  module  ratings  were  used  for  utility  worker  trainees.  Both 
modules  contained  work  sample  performanct'  tests  oi'  iilentification  pi'oblems 
on : 

(1)  Carpenter  tool  maintenance;  i e , the  sharpening  and  jointing  id' 
hand  saws . 


TABLE  7.  RELATION  OF  EXTERNAL  CRITERIA  TO  TEST  ITEMS 


Criteria 


Siimilation  Test  Iteni(s) 


Handsaw  sharpening  and  jointing 
Construction  layout 
Material  identificatiorr 


1,  2,  3,  and  9 
4,  5,  6,  7,  and  8 
10 


(2)  Material  identification;  i.e.,  proper  lumber,  nails,  etc.,  for  the  job. 

(3)  Construction  layout;  i.e.,  rudimentary  problems  in  determining 
proper  lengths  of  lumber  and  proper  angles . 

Table  7 shows  the  criteria/test  item  correspondence.  However,  the 
training  brigade  maintained  only  the  overall  module  ratings,  so  that  perform- 
ance ratings  on  item-specific  criteria  could  not  be  obtained. 

For  Unit  2,  the  "Building  Construction"  module  ratings  were  used  and 
the  material  identification  and  construction  layout  portions  appeared  to  offer 
adequate  correspondence. 

A combination  of  "Carpentry"  and  "Masonry"  module  ratings  were  used 
for  utility  worker  trainees. 

Because  the  carpenter  group  had  no  "Masonry"  module.  Unit  3 (Placing 
and  Finishing  Concrete)  was  administered  only  to  the  utility  workers  group. 
The  ratings  on  the  "Masonry"  module  were  used  as  criteria.  The  module  test 
included  the  actual  placing  and  finishing  of  a small  concrete  slab  as  well  as 
oral  tpiestions  on  when  and  how  to  vibrate  concrete. 

The  data  obtained  consisted  of  numerical  ratings  for  each  trainee  in  each 
course  module,  as  well  as  the  number  of  repetitions  of  the  module.  In  identi- 
fying criterion  ratings,  the  last  rating  obtained  by  the  trainee  was  used.  For 
example,  one  trainee  took  a module  three  times  and  obtained  ratings  of  47, 
30,  and  60.  Sixty  was  used  as  the  criterion  score.  It  was  reasoned  that  this 
was  a truer  index  of  his  performance  than  either  an  earlier  rating  or  an 
average  of  his  ratings. 


TABLES.  DESCRIPTION  OF 
EXPERIMENTAL  GROUPS  AND  CRITERIA 


Test  Unit 

Gioiip 

n 

Criteiia 

Unit  1; 

Hand  Tool  Characteiistics 

Caipenteis 

24 

Building  Constiuction  Module 

and  Material  Piepaiation 

Utility  Workers 

23 

Caipontry  Module 

Unit  2: 

Electing  Wall  Forms 

Caipenteis 

24 

Building  Construction  Module 

Utility  Woikeis 

23 

'j  (Carpentry  8i  Masonry  Modules) 

Unit  3 

Placing  and  Finishing 

Utility  Workeis 

23 

Masonry  Module 

Concrete 

'Applies  to  both  sitnuLition  (television)  jind  wntten  tests. 


The  expt'rimental  design  also  called  for  correlating  the  simulation  test 
with  a written  SQT'  component.  In  the  absence  of  the  written  component,  the 
research  team  constructed  a written  test  which  matched  the  simulation  test  on 
a task-for-task  level,  but  not  a critical  element-for-critical  element  level. 

SELECTION  OF  MATCHED  GROUPS 

because  of  the  small  number  of  subjects  available  for  this  study.  (Set' 
Table  8)  "matched"  groups  of  carpenter  and  utility  worker  trainees  were  used 
rather  than  random  samples  from  the  available  population.  One  group  re- 
ceived the  simulation  test  followed  by  the  written  test,  while  the  other 
received  the  written  test  followed  by  the  audio-visual  test. 

The  matching  variable  selected  for  the  carpenter  trainees  was  their 
rating  on  the  building  Construction  module.  The  matching  variable  selei-ted 
for  the  utility  worker  trainees  was  their  rating  on  the  ("arpentry  module 

Kach  group  (i.e.,  carpenters  and  utility  workers)  was  ranked  with 
regard  to  the  matching  variable  and  then  assigned  to  givups  A and  b.  as 
follows:  Ab,  bA,  Ab , bA,  etc.  Thus  for  every  pair,  the  higher  was  alter- 
nately assigned  to  groups  A and  b . 


TEST  ADMINISTRATION  PROCEDURES 


The  tests  were  administered  to  the  carpenter  trainees  in  the  morning  and 
the  utility  worker  trainees  in  the  afternoon. 

Each  group  reported  to  the  classrooms  that  were  used  for  simulation  test 
administration.  Students  were  assigned  to  one  of  the  two  classrooms.  One  of 
the  two  test  administrators  stayed  with  each  group. 

The  group  taking  the  simulation  test  was  told  that  it  was  an  experimental 
test,  and  that  their  scores  would  not  be  given  to  their  units.  They  were 
asked  to  do  their  best,  and  to  guess  at  the  answers  they  did  not  know. 

Answer  Sheets  (Appendix  C)  and  a diagram  of  the  wall  form  (Figure  1. 
Appendix  D),  were  distributed. 

Similar  instructions  were  given  to  the  group  taking  the  written  test 
(Appendix  D).  A separate  answer  sheet  was  not  required,  since  examinees 
circled  their  answers  in  the  test  booklet.  Since  the  simulation  test  took 
longer  to  administer,  students  completing  the  written  test  were  allowed  to 
walk  outside  of  the  building,  while  waiting  to  exchange  places  with  the  simu- 
lation test  group. 

As  soon  as  the  simulation  test  was  over,  the  groups  exchanged  places 
and  each  was  given  the  alternate  test. 

POOLING  OF  TEST  DATA 

Each  test  was  administered  to  both  groups  (i.e.,  carpenter  trainees  and 
utility  worker  trainees).  In  addition,  each  group  was  divided  into  two  sub- 
groups with  regard  to  testing  sequence  (i.e.,  one  received  the  simulation  test 
followed  by  the  written  test,  and  the  other  which  received  the  tests  in  re- 
verse order). 

To  obtain  a larger  n for  the  validation  process,  it  was  desirable  to  pool 
the  test  results  obtained  for  the  two  groups  of  trainees  and  the  two  orders  of 
presentation.  The  results  of  a two-factor  analysis  of  variance,  using  total 
test  scores,  is  shown  in  Tables  9 and  10.  It  can  be  seen  that  order  of 
presentation  (sequence  effects),  group  membership,  as  well  as  the  interaction 
between  the  two  factors  did  not  result  in  significant  variance.  Accordingly, 
data  from  both  groups  as  well  as  from  both  orders  of  presentation  was 
pooled . 


TABLE  9.  ANALYSIS  OF  VARIANCE 
FOR  SIMULATION  (TELEVISION)  TEST  SCORES 


Source 

df 

MS 

F 

P 

Order  of 

Presentation 

1 

26.01 

2.29 

Ns 

Carpenters  versus 
UtilitytWorkers 

1 

27.13 

2.39 

Ns 

Interaction 

1 

14.36 

1.26 

Ns 

Within  Groups 

43 

11.37 

— 

TABLE  10.  ANALYSIS  OF 
VARIANCE  FOR  WRITTEN  TEST  SCORES 

Source 

P_ 

Order  of 

Presentation 

1 

5.73 

1.01 

Ns 

Carpenters  versus 
Utility  Workers 

1 

22.18 

3.90 

Ns 

Interaction 

1 

7.55 

1.33 

Ns 

Within  Groups 

43 

5.68 

— 

.. 

Another  source  of  evidence  for  the  comparability  of  the  two  groups  was 
found  by  an  inspection  of  the  AFQT  scores  of  the  trainees.  The  average 
AFQT  score  for  the  carpenter  trainees  was  57.45  with  an  S.D.  of  20.86.  For 
the  utility  worker  trainees,  the  average  AFQT  score  was  48.90  with  an  S.D. 
of  24.49.  The  difference  between  these  group  averages  was  8.55  and  a test 
for  the  mean  differences  yielded  t = 1.26,  which  was  not  significant. 
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TABLE  11.  MEANS  AND  STANDARD 
DEVIATIONS  OF  RATING  SCORES  ON  CRITERION 
MODULES  BY  ASSOCIATED  SIMULATION  TEST  UNIT  AND  GROUP 


Simulation 

Test  Unit  Group  n Criterion  Module  Ratings 


1 

Carpenters  (A)* 

10 

Building  Construction 

87.80 

8.34 

Utility  Workers  (A) 

12 

Carpentry 

77.50 

16.03 

Carpenters  (B)* 

14 

Building  Construction 

84.36 

10.45 

Utility  Workers  (B) 

11 

Carpentry 

82.73 

12.32 

2 

Carpenters  (A) 

10 

Building  Construction 

87.80 

8.34 

Utility  Workers  (A) 

12 

(Carpentry  + Masonry) 

81.83 

15.63 

Carpenters  (B) 

14 

Building  Construction 

84.36 

10.45 

Utility  Workers  (B) 

11 

(Carpentry  + Masonry) 

86.55 

10.88 

3 

Utility  Workers  (A) 

12 

Masonry 

85.83 

16.76 

Utility  Workers  (B) 

11 

Masonry 

90.00 

14.14 

NOTE: 

Group  A were  given  the  simulation 

test  first 

Group  B were  given  the  written  test  first 

PERFORMANCE  ON  COURSE  MODULES 

Ratings  on  course  modules  were  used  as  the  performance  criterion  in 
validating  the  simulation  test.  Table  11  presents  the  means  and  standard 
deviations  of  the  distributions  of  ratings  for  the  samples  of  carpenters  and 
utility  workers  relative  to:  (1)  the  simulation  test  unit  associated  with  the 
specified  modules,  and  (2)  the  testing  sequence  for  the  two  groups  of  exam- 
inees. The  data  in  Table  11  are  interpreted  as  follows;  (1)  mean  perform- 
ance ratings  on  each  of  the  three  criterion  modules  (i.e.,  building  construc- 
tion, carpentry,  and  masonry)  were  quite  high,  and  (2)  the  examinees  were 
still  somewhat  heterogeneous  in  their  performance  levels  as  evidenced  by  the 
amount  of  variance  associated  with  the  means. 


TABLE  12.  MEANS  AND  STANDARD 
DEVIATIONS  OF  PERCENTAGES  OF  CORRECT  RESPONSES 
ON  THE  WRITTEN  TEST  FOR  CARPENTERS  AND  UTILITY  WORKERS 


Utility 


Carpenters 

Workers 

Pooled 

Test  Unit 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

1 

53 

16 

43 

19 

49 

18 

2 

50 

21 

42 

18 

46 

20 

3 

47 

19 

49 

15 

48 

12 

Total  Score 

51 

12 

45 

11 

48 

12 

PERFORMANCE  DATA  ON  WRITTEN  TESTS 

T'ho  writtt'n  tost  was  usod  as  a basis  for  assessing;  the  relative  value  of 
written  or  simulation  tests.  Examinee  performance  on  the  written  test  is 
summarized  in  Table  12.  The  means  of  the  percentages  of  correct  responses 
on  each  of  the  three  units  are  quite  low.  indicating;  that  the  test  was  fairly 
difficult.  The  mag^nitude  of  the  standard  deviations  sug^gests  a considerable 
rang:e  in  examinee  knowledgje  levels . 

PERFORMANCE  DATA  ON  THE  SIMULATION  TEST 

The  means  and  standard  deviation  of  the  distributions  of  test  scores 
(percentages  of  correct  answers  1 offered  by  the  carpenter  and  utility  worker 
ffroups  are  g-iven  in  Table  13.  These  means  are  close  to  the  50  percent, 
which  from  a psychometric  point  of  view  is  good  since  it  offers  the  greatest 
possibility  of  maximum  item  discrimination  (Ebel,  1972).  From  the  Army's 
perspective,  these  tests  are  somewhat  too  difficult.  The  magnitude  of  the 
standard  deviations  suggests  a considerable  range  in  exiuiiinee  abilities. 

UNIT  AND  OVERALL  TEST  VALIDITY 

The  estimation  of  test  validity  for  both  the  simulation  test  and  the  writ- 
ten test  was  accomplished  by  correlating  unit  and  overall  test  scores  with 
selected  module  ratings.  The  modules  that  were  selected  for  validating  each 
unit  and  the  associated  rationale  have  been  identified  earlier  in  this  report. 

.11 
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TABLE  13.  MEANS  AND  STANDARD  DEVIATIONS 
OF  PERCENTAGES  OF  CORRECT  RESPONSES  ON  THE 
SIMULATION  TEST  FOR  CARPENTERS  AND  UTILITY  WORKERS 


Utility 

Carpenter  Workers  Pooled 

(n  = 24)  (n  = 23)  (n  = 47) 


Test  Unit 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

1 

54 

23 

59 

17 

56 

20 

2 

44 

19 

51 

16 

48 

18 

3 

58 

21 

63 

13 

61 

18 

Total  Score 

52 

15 

57 

10 

54 

13 

The  correlations  are  presented  in  Table  14.  Both  the  simulation  test  and 
the  written  test  provided  significant  correlations  with  the  respective  criteria 
for  Unit  1 and  for  the  test  as  a whole.  The  contribution  of  the  variance  of 
Units  2 and  3 for  the  respective  tests  is  minimal,  so  it  must  be  concluded 
that  only  Unit  1 is  providing  variance  to  the  test  as  a whole. 

These  results  seem  to  represent  some  improvement  over  Cockrell's  (1976) 
results.  His  overall  correlation  of  the  written  test  was  .329  (p  < .01),  but 
his  overall  correlation  of  the  simulation  test  with  his  criterion  was  only 
.235  (NS). 

However,  it  must  still  be  acknowledged  that  the  magnitude  of  the  corre- 
lations indicate  that  both  tests  were  of  moderately  low  validity . 

ASSESSMENT  OF  ATTITUDES  TOWARD  SIMULATION 

Examinee  attitudes  toward  simulation  testing  were  assessed  via  a question- 
naire . 

In  developing  the  questionnaire,  the  research  team  adapted  items  found 
in  the  Procedures  for  Validating  Skill  Qualification  Tests  (Hirshfeld,  Young 
and  Maier,  1975),  informal  questionnaires  obtained  from  the  Aimy  serv'ice 
schools,  as  well  as  questionnaires  found  in  academic  literature.  The  question- 
naire contained  both  structur  (scaled)  items  and  open-end  items.  It  was 
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TABLE  14.  CORRELATION  OF  SIMULATION  AND 

WRITTEN  TEST  SCORES  WITH  RATINGS  ON  CRITERION  MODULES 

Test  and 

Unit 

Pearson  r 

n 

P 

Simulation  Test 

Unit  1 

.427 

47 

.01 

Unit  2 

.145 

47 

NS 

Unit  3 

.068* 

47 

NS 

Total  Score 

.350 

47 

.02 

Written  Test 

Unit  1 

.448 

47 

.01 

Unit  2 

.021 

47 

NS 

Unit  3 

.182* 

47 

NS 

Total  Score 

.430 

47 

.01 

•Correlation  based  upon  utility  worker  trainees  only, 

since  criterion 

(Masonry  Module  Scores)  was 

not  available  for  carpentry  trainees. 

designed  to  assess  and 

contrast  attitudes 

toward  both  the  simulation  and 

written  tests. 

STRUCTURED  tTEMS 

These  items  were  constructed  to  provide  a scaled  response  on  a four- 
point  scale.  The  data  were  recorded  separately  for  examinees  who  were  tested 
in  the  morning  versus  those  tested  in  the  afternoon  because  of  the  differ- 
ences reported  by  Cockrell  (1976). 

The  data  presented  in  Table  15  reveals  that  overall,  the  examinees  felt 
that  the  A/V  simulation  test  was  slightly  fairer,  slightly  more  interesting,  and 
very  slightly  less  difficult  than  the  written  test.  The  data  further  reveal 
that  on  seven  of  eight  items  (Items  1,  2,  4,  5,  6,  7,  9,  10)  in  which  one  end 
of  the  scale  represented  a positive  feeling,  the  average  response  (mean)  fell 
between  mid-scale  and  the  positive  end.  The  one  item  in  which  the  mean  fell 
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TABLE  15.  SCALED  QUESTIONNAIRE  DATA 


To  what  extent  was  either  test  a fair  measure  of  your  ability  to  perform  in  your  MOS? 
Television  Test 


1  Extremely  fair 

2  Very  fair 

3  Somewhat  fair 

4 Not  fair 


X = 2.36 


2.  How  interesting  was  either  test? 

Television  Test 
J Extremely  interesting 

2  Very  interesting 

3  Somewhat  interesting 

4  Not  interesting  IT  = 2 

3.  How  difficult  was  either  test? 

Television  Test 

1  Extremely  difficult 

2  Very  difficult  ~X  = 3 

3  Somewhat  difficult 

4  Not  difficult 

4.  Overall,  to  what  extent  were  the 
television  test? 


Written  Test 
J Extremely  fair 

2  Very  fair 

3  Somewhat  fair 

4 Not  fair 


Written  Test 


X = 2.64 


44 


1  Extremely  interesting 

2  Very  interesting 

3  Somewhat  interesting 

4  Not  interesting  ”X  = 3 1 1 


27 


Written  Test 
J Extremely  difficult 

2  Very  difficult 

3  Somewhat  difficult 

4 Not  difficult 


X = 3.24 


visuals  (pictures,  graphics,  titles,  etc.)  clear  in  the 


1  Extremely  clear 

2  Very  clear 

3 Somewhat  clear 

4 Not  clear 


X = 2.28 


5.  In  the  case  of  television  test,  to  what  extent  was  the  narration  easy  to  understand? 

1  Extremely  easy  to  understand 

2  Very  easy  to  understand  "v"-  i 

3 Somewhat  easy  to  understand  ^ ~ 

4 Not  easy  to  understand 
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TABLE  15.  SCALED  QUESTIONNAIRE  DATA  (Continued) 


6,  In  the  case  of  the  television  test,  to  what  extent  was  the  answer  easy  to  use? 


J Extremely  easy  to  use 

2  Very  easy  to  use 

3  Somewhat  easy  to  use 

4  Not  easy  to  use 


X = 2.08 


7.  Overall,  did  you  have  enough  time  to  answer  the  questions  to  the  televfsion  test? 


1  More  than  enough  time 

2  Enough  time 

3  Barely  enough  time 

4  Not  enough  time 


X = 2 34 


What  is  your  feeling  about  the  overall  pace  (rate  of  presentation)  of  the  television 
test? 


J The  pace  was  much  too  slow 

2  The  pace  somewhat  too  slow 

3  The  pace  was  somewhat  too  fast 

4 The  pace  was  much  too  fast 


X = 2 49 


What  is  your  feeling  about  the  overall  selection  of  items  (situations)  for  the  television 
test? 


1  The  items  were  extremely  well  chosen 

2  Jhe  items  were  very  well  chosen 

3  The  items  were  fairly  well  chosen 

4  The  items  were  poorly  chosen 


X = 2.71 


10.  From  where  you  are  sitting,  how  well  were  you  able  to  see  the  television  screen? 


J Extremely  well 

2  Very  well 

3  Fairly  well 

4 Not  well 


X = 1.60 
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on  the  negative  side  of  the  midpoint  (item  9)  pertained  to  the  selection  of  test 
items.  Item  8 referred  to  the  pace,  and  the  responses  were  close  to  the 
midpoint,  indicating  a pace  that  was  neither  too  fast  nor  too  slow.  The  tenth 
item  (item  3)  referred  to  test  difficulty,  and  responses  indicated  that  the  test 
was  judged  to  be  somewhat  difficult. 

OPEN  END  ITEMS 

The  constructed  items  yielded  only  one  consistent  finding,  namely  that 
Unit  2 was  a little  confusing.  Some  representative  comments  to  the  various 
open-end  questions  are  given  below: 

(1)  "Can  you  recall  any  specific  items  (situations)  in  the  television  test 

that  are  confusing?  If  so,  describe  the  item(s)  in  a few  words". 

(a)  I think  that  in  Unit  #2  they  went  too  fast.  You'd  write  on 
answer  sheet,  look  up,  and  the  next  question  was  almost  over. 

(b)  On  Unit  #2,  the  skipping  around  was  a little  confusing.  Also 
looking  for  safety  hazards  was  hectic. 

(c)  The  building  of  the  wall  form  - was  extremely  hard  to  follow 
the  procedures. 

(2)  "Do  you  have  any  additional  comments  on  the  television  test"? 

(a)  I thought  it  was  a good  test.  1 think  those  kind  of  tests 
would  be  alright! 

(b)  There  wasn't  enough  verbal  instruction  during  the  test. 

(c)  Many  of  the  pictures  (alternatives--Ed . ) looked  the  same. 

(3)  "Can  you  recall  any  specific  items  in  the  written  test  that  were 

confusing?  If  so,  describe  the  item(s)  in  a few  words". 

(a)  The  parts  about  filing  a saw. 

(b)  The  diagram  of  the  wall  form. 

(4)  "Do  you  have  any  additional  comments  on  the  written  test"? 

(a)  The  written  test  to  me  wasn't  clear  enough. 

(b)  It  was  a lot  easier  to  understand  than  the  television  and  you 
could  work  at  your  own  speed. 

(c)  I think  it  is  better  to  show  the  TV  because  of  the  people  that 
can't  read. 
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COMPARATIVE  COSTS  OF  VIDEOTAPE  SIMULATION  TESTS 


It  was  that  tlu-  actual  cost.-,  of  ilt'Vt'loping- . valtdatuij; . anil 

ailininisliM'ing'  the  pi'ololypr  test  woulil  b<‘  ci'iupai'oil  with  the  ai'tual  costs  ol 
il('vclopin>;' , valul.it iii^;' , aiul  a^^lmnistcI■in^;■  the  same  test  in  a hamls-on  pei’foc- 
manci'  aiul  wcitten  fi'cmat  .As  previoin.ly  staleil.  no  h.aiuls-on  or  written 
tests  e.xisleil  ami  the  Army  was  unable  to  supply  firm  data  as  to  development 
and  aiiminisl  rat  ion  costs  of  any  .SQ’l’  component  Noi’  wai.  ci'st  data  available 
from  the  only  otlu'r  two  known  ti'levision  simulation  tests  T’herefori' . cert.iin 
costs  pertaining;'  to  the  development  and  administration  of  hands-on  and  writ- 
ten versions  of  tin-  (U'ototype  simulation  lest  were  estimated  by  the  research 
te.im's  test  development  staff  who  recenily  completi'd  an  .Sc^T'  for  the  Ord- 
nance School  at  Aberdeen  I’lovinj;  llrounds.  MU  In  addition,  because  the 
simulatii'n  ti'st  was  a pridotypi'  and  not  inteiuied  fi'r  ai'tual  use.  the  ailmini- 
stration  costs  for  it  were  estimated  The  assumptions  undiu'  which  thise 
I'stiiiiates  w<>re  made  are  stated  as  follows. 

Assuniptiuns 

t,  n One  vidi'otapi'd  test  is  ^iveii  to  a ma.vimuiii  of  I.S  soldier;,  in  one 
administration,  the  test  is  till  minutes  lonj;  and  requires  1-12 
manhours  to  administer 

021  t)ne  hands-on  test  is  {fiven  to  one  soldier  in  oni'  administration,  tin- 
test  may  take  up  to  two  hours  and  requires  one  rati-r  to  ailiiiiinster 

OO  One  written  test  is  ^'iven  to  .1  maximum  of  Ml  soldiers  in  one  admin- 
istration. The  test  is  -lH  minutes  lont;'  and  requires  1 manhour  to 

administer  Material  cost  (paper!  for  this  administration  is  $2h  00 

(11  Materi.'ils  used  in  tin-  hands-on  ti-st  . i t-  , liimbi-r,  concrete,  etc  . 
are  considered  as  consumables  in  th;it  they  would  not  be  re-used 
for  testiiqj'  piirpos<-s 

(!>)  l-'or  the  purposi-s  of  this  ;in;ilysis,  each  test  site  would  n-ceive  two 
video  cassette  copii-s  (one  for  back-up!  of  the  test  t>ne  cassette 
m;iy  be  phiyed  1>0  tiiin-s  before  its  quality  is  dt'>;r;ided  (This  is  ;i 
const'i'vat ive  «'stim;iti'  ! Therefon-,  oin-  i-assette  I'ould  i-oiu'eiv.ibly 
ti'st  TliO  soldii-rs  (!i0  replays  limes  lf>  soldiers! 


to 


(6)  C?osts  for  administering  all  three  tests  are  linear;  i.e  , the  costs  of 
administering  the  same  test  to  the  same  number  of  soldiers  will  be 
the  same  for  the  first  test  and  the  1.000th  test,  provided,  of 
course,  assumptions  1 through  4 are  met. 

(7)  Because  dollar  juiiounts  seemed  the  most  appropriate  common  denomi- 
nator, it  was  necessary  to  ascribe  an  hourly  rate  to  a "typical" 
N(X>  scorer/test  administrator.  For  puiposes  of  this  analysis,  an 
K-6  (Staff  Sergeant)  was  chosen,  with  six  years  service,  no  depen- 
dents and  no  proficiency  pay.  His  hourly  rate  was  estimated  as 
follows : 


BASK  BAY 

ALLOW ANCK  FOR  QHAHTKHS 
ALLOWANCK  FOR  SUBSISTKNCK 
(H.OTHINO  ALLOWANCK 


OO/mo. 
$1 17.00/mo. 
$ 79..‘)0/mo. 
$ 7. 00 /mo. 


TOTAL 
I'KR  BOHR 


$Sb0 . 00/mo 
$ O.OO/hr. 


Military  pay  is  based  on  a OO-day  month  and  there  is  no  stated  standard 
work  day.  With  this  in  mind,  the  typical  NCO  was  arbitrarily  assigned  a 
10-hour  work  day  with  allowance  for  a minimum  of  four  non-work  days  per 
:l0-day  peruul.  This  reduces  to  260  productive  hours  per  pay  period,  at 
$0  30 /hour . 

Table  16  shows  actual  expenditures  of  professional  manhours  for  devel- 
opment and  validation  of  the  videotaped  test  and  estimated  expenditures  for 
the  hands-on  ami  written  tests. 

Costs  involved  in  task  analysis,  task  selection,  coordination,  media 
selection,  and  the  like  are  not  included  as  it  is  assumed  these  costs  would  be 
roughly  the  same  regardless  of  test  format . I’ravel  costs  are  not  considered 
as  they  are  extrinsic  to  the  development  process  and  would  probably  not 
occur  in  an  operational  (as  opposed  to  a research)  mode. 

Material  costs  for  the  videotaped  test  are  as  shown  in  I'.able  17. 

Kquipment  costs  are  considered  as  sunken  costs  and  therefore  not  rele- 
vant to  the  analysis.  The  Army  is  upgrading  and  expanding  its  videotape 
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TABLE  16.  DEVELOPMENT  AND  VALIDATION  EXPENDITURES 


Professional  Manhours 


Task 

Videotape 

Hands-On* 

Written* 

Development 

848 

120 

16 

Validation 

80 

40 

8 

Total 

928 

160 

24 

•Estimated 


production  capability  and  the  equipment  and  operating  personnel  are  presently 
used  in  the  production  of  training  and  command  information  materials.  There- 
fore, the  equipment  costs  in  production  of  videotaped  tests  are  really  oppor- 
tunity costs;  that  is,  the  cost/benefit  of  producing  tests  versus  training 
materials. 

The  alternative  uses  of  existing  capital  equipment  are  policy  decisions 
and  beyond  the  scope  of  this  report.  It  will  be  demonstrated,  however,  that 
the  relatively  low  administration  costs  of  videotaped  tests  make  it  a feasible 
test  delivery  system  when  compared  to  the  costs  of  administering  similar 
hands-on  tests.  The  savings  in  administration  outweigh  the  admittedly  high 
costs  of  development  and  production. 

Material  costs  used  in  validation  of  the  hands-on  test  would  be  the  same 
(less  the  expense  of  the  videotape),  or  about  $200.00.  As  can  be  readily  seen 
from  the  preceding  data,  development  costs  of  the  videotaped  tests  are  signi- 
ficantly greater  than  those  of  the  hands-on  version  (848  professional  man- 
hours compared  to  120  professional  manhours).  This  supports  Cockrell's 
observation  (page  13).  However,  administration  costs  demonstrate  the  feasi- 
bility of  the  videotaped  version. 

Costs  per  soldier  incurred  in  administering  the  hands-on  test 
(assumption  2)  are  computed  as  follows; 
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TABLE  17.  VIDEOTAPED  TEST  MATERIAL  COST  (DEVELOPMENT) 


Concrete 

$ 25.00 

Lumber 

$175.00 

Videotape  Cassettes 

$500.00 

$700.00 

PER  SOLDIER  COSTS  OF  HANDS-ON  TEST  ADMINISTRATION  = 

(MATERIAL  COSTS)  + (NCO  HOURS  X $3.30) 

NUMBER  OF  EXAMINEES  TESTED  IN  ONE  ADMINISTRATION 


Based  on  the  estimates,  the  hands-on  test  would  cost 
or  $206.60  per  soldier  tested. 


($200)  + (2  X $3.30) 
1 


The  same  formula  is  used  for  computing  the  per  soldier  costs  of  adminis- 
tering the  written  test  (assumption  3).  Thus,  the  test  cost 
$3.30)  slightly  less  than  $0.57  per  soldier  tested. 


On  the  other  hand,  the  cost  per  soldier  of  administering  the  videotaped 
version  (assumption  1)  can  be  stated  as  the  cost  of  the  tapes  divided  by  the 
total  number  of  examinees  (up  to  750,  assumption  5),  plus  the  total  NCO 
hours  expressed  in  dollars,  also  divided  by  the  total  number  of  examinees. 
The  NCO  dollar  fig^ire  is  computed  by  dividing  the  total  number  of  examinees 
by  15  (assumption  1),  counting  the  remainder  as  the  next  whole  unit,  and 
multiplying  that  figure  by  $3.30  (assumption  7). 


A2 


The  formula  can  be  stated  as  follows; 


PER  SOLDIER  COST  OF  VIDEOTAPED  TEST  ADMINISTRATION  = 


COST  PER  TAPE  X NUMBER  OF  TAPES 
NUMBER  OF  EXAMINEES 


'/number  of\» 

I EXAMINEES  j x 


$3.30 


♦Rounded  up  to  whole  number. 

Based  on  the  estimates,  then,  the  per  soldier  cost  of  the  prototype  test 
administration  would  be  (number  of  examinees  = 47): 

^ X $3.30^or  $1.28  + $13.20,  or  $14.48  per  soldier  tested. 

The  comparative  test  administration  costs  per  soldier  are: 

Hands-on  test  $206.60 

Videotaped  test  $ 14.48 

Written  test  $ 0.57 

As  mentioned  earlier,  the  videotape  production  equipment  costs  have  not 
been  considered  here.  However,  it  was  felt  that  an  equipment  configTaration 
which  is  considered  as  the  minimum  necessary  to  produce  the  prototype 
should  be  described.  The  equipment  detailed  in  Table  18  could  be  purchased 
for  approximately  $60,000.00. 

The  preceding  discussion  deals,  of  course,  only  with  the  cost  advantages 
of  the  videotaped  simulation  test.  In  this  light,  it  supports  Cockrell's  conten- 
tion, "The  tests  can  be  produced  at  a reasonable  cost  ..."  (page  16)  and 
that,  "television  testing  is  far  less  cumbersome  and  costly  than  hands-on 
testing"  (page  13).  It  is  recognized  that  decisions  based  purely  on  monetary 
cost  savings  ignore  the  many  other  aspects  that  contribute  to  the  value  of  a 
given  alternative.  If  the  simulation  test  does  not  measure  knowledge  and 
skills  with  acceptable  accuracy,  the  price  may  be  right,  but  its  value  is 
questionable. 


/$30  X 2 
\ 47 


TABLE  18.  VIDEOTAPED  PRODUCTION  EQUIPMENT  CONFIGURATION 


PORTABLE  EQUIPMENT  QUANTITY 

Portable  vidicon  color  cameras  with  tripods  2 

Portable  video  cassette  color  recorder  with  1 

microphone 

Portable  lighting  kit  2 

STUDIO  EQUIPMENT 

Video  cassette  recorder/p  layers  with  edit  capability  2 

Edit  controller  1 

Special  effects  console  1 

Time  base  corrector  1 

Monitor  (color)  2 

Audio  mixer  1 

Audio  tape  recorder  1 

LESSONS  LEARNED 

The  following  is  a general  discussion  of  judgments  and  observations  made 
by  the  research  team  during  the  course  of  the  simulation  test  construction 
process . 

LIMITATIONS  AFFECTING  THE  VALIDATION  OF  THE  SIMULATION  TEST 

First,  test  items  within  units  were  not  adequately  revised  following 
review  of  the  completed  prototype  test  by  subject  matter  experts  at  Fori 
Leonard  Wood.  This  was  due  to  scheduling  problems  discussed  earlier.  On 
some  items,  there  was  disagreement  among  the  experts  as  to  what  the  pre- 
scribed doctrine  was,  even  when  the  doctrine  was  spelled  out  in  the  relevant 
technical  manuals.  This  lack  of  agreement  among  technical  experts  seems  to 


extend  to  the  construction  of  the  standard  SQTs  as  well.  It  is  exacerbated 
somewhat  when  television  is  the  testing  mode  because  making  changes  to  vid- 
eotape is  more  cumbersome  than  making  changes  to  written  tests  or  hands-on 
test  scoring  procedures.  The  "lesson"  seems  to  be,  to  insist  on  a more 
intensive  and  higher  level  review  of  the  test  in  its  storyboard  and  scripting 
stages . 

A second  factor  pertains  only  to  Unit  2 of  the  simulation  test.  This 
unit  used  a relatively  unalerted  response  format . in  which  the  examinee 
had  to  record  an  event  which  might  occur  anytime  during  a unit  of  up  to  two 
minutes.  The  exjuninees  were  not  familiar  with  this  response  format, 
and  they  reported  some  confusion  in  their  questionnaire  responses.  The 

testing  session  did  not  include  any  example  responses  for  the  Unit  2 

response  format.  Such  warm-up  may  have  alleviated  the  problem.  It  is 
noted  that  Cockrell  (1976)  indicated  the  need  for  examinee  practice  in 

responding  to  televised  test  items. 

Third,  and  perhaps  most  importantly,  the  criterion  measure  was  only 
indirectly  related  to  the  simulation  test  item.  This  factor  alone  should 
immediately  be  expected  to  reduce  the  correlation  to  a moderate  level  at 
most.  The  experimental  design  called  for  a validation  on  an  item-by-item 

basis . Since  appropriate  performance  measures  were  not  supplied  by  the 
Army,  the  simulation  test  was  validated  against  a measure  of  perfomance 
from  a large  training  module.  The  module  may  or  may  not  have  included  the 
specific  skills  which  were  tested  in  the  simulation  test,  and  the  observations 
may  have  been  made  up  to  a month  prior  to  the  administration  of  the  simula- 
tion test. 

PROBLEMS  INHERENT  IN  PRESENT  APPLICATION  OF  A/V  SIMULATION  TESTS 

The  overall  purpose  of  this  research  effort  was  to  validate  a promising 
method  of  audio-visual  simulation  to  test  perceptual  job  content  in  a real 
world  application.  Previous  discussion  has  identified  a few  pi'oblems  encoun- 
tered, most  of  which  were  not  inherent  in  the  situation.  At  this  time,  it  is 
appropriate  to  specify  problems  which  would  have  occurred  even  if  logistic 
problems  such  as  scheduling  and  availability  of  subject  matter  experts  had 
not  occurred.  These  problems  are  addressed  because  they  should  be  con- 
sidered in  other  attempts  to  specify  appropriate  applications  of  A/V  simula- 
tion testing. 


PERCEPTUAL  CONTENT  IN  JOB 


The  present  application,  by  design,  was  limited  to  perceptual  or  percep- 
tual related  psychomotor  task  components.  The  first  obvious  requirement  for 
this  application  was  that  there  be  appropriate  perceptual  or  perceptual  related 
tasks  to  test.  In  fact,  few  tasks  in  the  levels  one  and  two  of  MOS  51B  were 
appropriate.  While  all  three  units  of  the  simulation  test  addressed  perceptual 
components  of  tasks,  it  is  recognized  that  for  Unit  1 in  particular,  cognition 
was  more  critical  to  accurate  performance  than  perception.  (Further  errors 
in  actual  task  performance  are  most  often  psychomotor).  For  example,  an 
incorrect  response  to  an  item  showing  four  positions  for  pointing  a hand  saw 
was  more  likely  related  to  not  "knowing"  the  right  position  than  to  not  "dis- 
criminating" the  alternative  positions.  In  MOS  51B,  critical  perceptual  tasks 
occur  more  at  the.  supervisory  level  where  there  are  inspections  of  work  in 
process  or  accomplished. 

SIMILARITY  OF  TEST  AND  JOB  RESPONSE 

In  a preceding  discussion  of  test  validity,  the  importance  of  response 
variables  was  noted.  In  the  present  application,  levels  1 and  2 tasks 
typically  involve  psychomotor  responses  and  the  responses  are  not  quite  as 
alerted  as  in  the  multiple -choice  format.  (The  response  of  the  supervisor 
which  is  often  a verbal  response,  would  not  be  as  discrepant.)  In  the  pres- 
ent application,  disparity  between  the  job  response  and  available  test  re- 
sponses presented  a problem. 

GENERAL  CONSIDERATIONS  CONCERNING  THE  USE  OF  TELEVISION  SIMULATION 


MOTIVATIONAL/ATTITUDINAL 

Videotape  seems  to  have  an  advantage  over  performance  and  paper-and- 
pencil  tests  insofar  as  its  acceptability  to  the  examinee  is  concerned.  This  is 
an  important  criterion  in  the  consideration  of  test  formats.  Bloom  (1970) 
concludes  that  student  measurement  can  have  both  positive  and  negative 
effects  and  that  the  person  being  evaluated  will  always  respond  to  evaluation 


in  terms  of  the  perceived  fairness.  This  perceived  fairness  is  enhanced 
through  television  because  the  test  developer  is  able  to  take  advantage  of  the 
"transfer  effect"  as  potential  examinees  are  already  highly  receptive  to  the 
medium.  Thus,  the  television  test  builds  on  habit  patterns  already  firmly 
established  in  the  examinee.  Cockrell  (1976)  argues  for  continued  interest  in 
the  use  of  television  as  the  stimulus  input  in  synthetic  performance  testing. 
He  lists  three  reasons  given  by  examinees  for  preferring  television  testing; 

"1.  Scoring  is  fairer  and  not  dependent  upon  the  whims  of  the  test 
administrator. 

2.  Testing  is  faster  and  not  so  drawn  out. 

3.  In  television  testing,  no  one  is  shouting  at  you  and  ordering 
you  around . " 

The  questionnaire  results  reported  both  favorable  attitudes  towards  the 
TV  test  and  more  favorable  attitudes  toward  it  than  toward  a written  test. 
Thus,  although  the  present  TV  application  was  justified  only  in  terms  of 
perceptual  test  content,  developers  of  SQTs  might  wish  to  consider  these 
other  factors. 

RESOLUTION/ACUITY 

When  judgments  are  based  on  fine  perceptual  discriminations,  such  as 
the  presence  or  absence  of  a light  water  sheen  on  concrete  or  minor  differ- 
ences in  the  color  or  grain  of  types  of  lumber , television  may  not  be  able  to 
faithfully  reproduce  these  visual  cues.  There  is  no  standard  picture  quality 
from  one  television  set  to  the  next;  hence,  there  is  no  assurance  that  visual 
stimuli  simulated  faithfully  on  a studio  monitor  will  not  be  obliterated  by  a set 
with  poorer  resolution  in  the  field . 

Because  TV  or  any  A/V  medium  which  may  be  selected  for  simulation  is 
limited  with  respect  to  the  stimuli  that  can  be  faithfully  reproduced,  it  is 
necessary  to  carefully  analyze  the  perceptual  content  before  deciding  that 
simulation  is  an  acceptable  testing  mode. 


COSTS  OF  AA/  SIMULATION  VERSUS  WRITTEN  AND  PERFORMANCE  TESTING 


The  costs  of  administering  a television  test  are  significantly  less  than 
those  of  a performance  test  and  only  slightly  higher  than  those  of  a written 
test.  The  major  costs  of  the  television  test  are  incurred  during  the  produc- 
tion stage.  These  costs  can  be  lessened  through  the  early  involvement  of 
media  experts  in  the  test  development  process,  so  that  exceptionally  costly  or 
time  consuming  segfments  can  be  kept  to  a minimum . Production  costs  can 
also  be  lessened  through  an  intensive  review  of  the  test  in  the  scripting 
stage,  so  that  editing  and  "reshooting"  time  is  minimized. 

SIMULATION  OF  DYNAMIC  TASKS 

Television  is  able  to  realistically  simulate  situations-  where  the  viewer  has 
to  make  judgments  based  on  his  observations  of  relatively  complex,  dynamic 
events  in  which  many  variables  are  changing.  The  television  portion  of  the 
Military  Policeman  SQT,  for  example,  has  a great  deal  of  face  validity  in  its 
presentation  of  many  such  situations  as  test  items.  Through  the  use  of 
television,  the  test  developers  were  able  to  simulate,  with  adequate  realism, 
the  scene  of  a burglary,  complete  with  scattered  pieces  of  evidence  which  the 
examinee  was  asked  to  identify  and  note  (the  same  response  as  required  on 
the  job).  Such  an  item  could  not  have  been  described  verbally  without 
extreme  overcuing.  Likewise,  the  MPs  were  able  to  simulate  a situation 
which  required  that  the  examinee  locate  a particular  suspect,  based  on  brief 
descriptions,  from  among  a moving  crowd  of  people,  whde  the  camera  moved 
through  a neighborhood  in  a patrol  car.  These  items  called  for  judgments 
based  almost  entirely  on  fine  perceptual  discriminations  and  demonstrated  the 
true  appropriateness  of  the  medium. 

Soldiers  at  higher,  supervisory  skUl  levels  must  frequently  make  judg- 
ments based  on  their  observations  of  dynamic  events,  such  as  the  crew  per- 
formance involved  in  many  construction  tasks.  At  the  higher  skill  levels, 
however,  many  times  the  proper  response  to  a given  stimulus  is  the  judg- 
ment. The  supervisor's  task  may  be  to  observe  a process  and  note  errors 
and  inconsistencies  in  the  process.  If  this  process  is  a dynamic  one,  where 
changes  in  the  states  of  the  variables  overlay  one  another  and  do  not  neces- 
sarily occur  in  a prescribed  sequence,  then  a dynamic  medium  such  as  tele- 
vision is  necessary  if  the  process  is  to  be  faithfully  simulated 
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I'ht'  few  truly  peiveptual  jiuignuents  maile  by  soldiers  at  skill  levels 
1 and  U.  however,  are  j^enerally  based  on  stimuli  whioh  are  statio  or  at 
h'ast  flow  in  an  easily  prt'dii'table  seiiuence.  This  statio,  or  sequential 
oharaoteristio  of  task  elements  at  the  lower  skill  levels,  makes  television 
less  appropriate  as  a test  medium  in  those  cases. 


CONCLUSIONS 


TASK  SELECTION  SIMULATION  PROCEDURES 

1.  The  application  of  the  procedures  enabled  the  selection  of  the  more 
appropriate  tasks  and  task  components  from  a specified  field  of  tasks  critical 
to  MOS  51 A and  51B. 

2.  Use  of  the  simulation  procedures  requires  a greater  expenditure  of 
human  resources  than  may  typically  be  present  in  a test  development  agency. 

APPLICABILITY  OF  AA/  SIMULATION 

1.  The  fundamental  question  of  the  applicability  and  validity  ot  A'V 
simulation  to  test  perceptual  content  was  not  conclusively  answered  because  of 
a number  of  problems  discussed  in  the  text  of  the  report . 

2.  The  use  of  television  as  a simulation  means,  strictly  for  testing  the 
perceptual  content  of  lower  skill  level  motor  tasks  such  as  those  within  the 
Carpentry  and  Masonry  MOS.  appears  somewhat  limited;  there  appears,  how- 
ever. to  be  a decided  favorable  attitudinal  bias,  on  the  part  of  the  test 
takers,  towards  television  testing. 
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Appendix  A 

TASK  SELECTION  PROCEDURES 


APPENDIX  A 


TASK  SELECTION  PROCEDURES 


INTRODUCTION 

The  ideal  task  for  use  as  a test  candidate  can  be  defined  as  one  wfhich 
% 

requires  the  application  of  every  key  and  essential  behavior  component  of 
tasks  within  the  MOS.  Such  a task  does  not  exist  and  if  one  was  created 
solely  for  test  purposes  it  would  lack  reality  and  continuity  as  it  would  likely 
differ  from  "real  world"  job  performance.  Existing  tasks  and  task  clusters 
must  then  be  examined  with  the  goal  of  identifying  those  which  most  closely 
approximate  the  "ideal" . Candidate  tasks  then  become  those  which  require 
the  application  of  the  greatest  number  of  separate,  distinct,  key.  and 
essential  behaviors  which  are  common  to  the  majority  of  tasks  within  the  MOS . 

DEVELOPMENT  OF  BEHAVIOR  DESIGNATORS 


To  select  the  candidate  tasks  at  a given  skill  level,  behavior  designators 
(explained  later)  are  used  to  identify  elements,  or  performance  steps,  within 
each  task.  A matrix  is  then  developed  to  identify  common  elements 
(behaviors)  which  cut  across  tasks  and  equipment.  An  example  of  this  matrix 
is  included  as  Figure  A-1,  and  should  be  studied  as  the  analyst  reads  the 
explanation  that  begins  with  Step  1 . The  matrix  is  constructed  by  foUowing 
the  algorithm  presented  as  Figure  A- 2.  In  this  algorithm,  action  steps  are 
enclosed  by  a circle  , questions  by  a diamond  , and  answers  by 


a square 


□ 


Each  step  is  explained  in  detail  following  this  introduction. 

In  general  the  matrix  contains  two  types  of  entries;  the  first  is  a listing 
of  all  critical  tasks  at  a given  skill  level  and  second  is  a listing  of  the 
behavior  designators  pertinent  to  the  tasks.  This  will  then  enable  you  to 
rank  order  the  tasks  as  candidates  for  inclusion  in  an  SQT. 


TASK  SELECTION  PROCESS 

Identify  skill  level.  Separate  tasks  by  skill  level  so 
that  only  one  level  will  be  considered  for  any  one  matrix. 


Identify  critical  tasks.  Prior  to  establishing^  commonality,  the 
importance  of  each  task  must  be  assessed.  Most  of  the  time,  the  documen- 
tation which  contains  the  task  performance  steps  will  also  indicate  the  criti- 
cality or  level  of  importance  of  the  task.  If  it  does  not,  at  least  two  subject 
matter  experts  should  assess  each  task  using  the  following  basic  guidance. 
"Two  major  classes  of  importance  are.  (1)  criticality  to  mission  accomplish- 
ment, based  on  expert  judgments,  and  (2)  performance  deficiencies  in  the 
field,  documented  by  field  data  demonstrating  weak  perfontiance.  Potential 
sources  of  data  include  Army  Training  & Evaluation  Program  (ARTEP)  results. 
Maintenance  Management  Center  (MMC)  data.  Equipment  Serviceability  Criteria 
(ESC)  reports,  Inspector  General  inspection  reports,  and  morning  reports". 

Should  the  critical  tasks  be  separated  and  grouped  by  functions? 
Some  MOSs  will  contain  so  large  a number  of  critical  tasks  that  some  way  will 
have  to  be  found  to  reduce  the  job  to  manageable  bites  of  say , 40  to  60  tasks 
per  matrix.  One  way  is  to  group  the  tasks  functionally ; for  instance,  tasks 
in  the  MOS  62F,  "Crane  Operator,"  can  logically  be  grouped  by  "Maintenance" 
functions  and  "Operations"  functions. 

YES.  A yes  answer  sin\ply  means  that  at  this  point,  you  should 
divide  the  tasks  functionally  so  that  you  will  be  able  to  construct  one  matrix 
for  each  functional  area.  Using  the  example  of  two  MOS  62F  tasks  which  are: 

(1)  Perform  operator's  maintenance  on  the  crawler  crane. 

(2)  Drive  the  truck  mounted  crane  between  job  sites. 

The  Job  Task  Summary  Sheet  (JTSS)  for  task  (1)  lists  18  separate 
performance  steps,  ranging  from  inspection  and  replacement  to  lubrication. 
These  are  clearly  preventive  maintenance  functions  as  the  behavior  desig- 
nators indicate.  The  JTSS  for  task  (2)  lists  six  separate  perfomance 
steps  such  as  positioning  the  boom,  retracting  outriggers,  and  starting  and 
stopping  the  crane.  These  are  clearly  operational  functions. 


Procedures  for  Validating  Skill  Qualification  Tests,  Stephen  K.Hitshfeld, 
Douglas  L.  Young,  & Milton  H.  Maier,  IJ.S.  Army  Kesearch  Institute  for  the 
Behavioral  and  Social  Sciences,  June  1476.  (Draft) 


After  all  tasks  have  been  grouped  by  functions,  you  would  proceed  to 
Step  (T) 

No.  Move  on  to  Step 

Should  the  critical  tasks  be  grouped  by  systems?  This  question 
is  asked  for  much  the  same  reasons  as  those  explained  in  Step 
example,  the  soldier's  manual  for  MOS  63C,  "Track  Vehicle  Mechanic"  shows 
some  290  tasks.  These  tasks  however,  can  be  grouped  by  several  major 
systems,  such  as.  Engine  & Ignition  System,  Cooling  System,  Fuel  System, 
Electrical  System,  Suspension  System,  etc.,  and  a separate  commonality 
matrix  may  be  constructed  for  each  system. 

|T1  YES.  A yes  answer  means  that  at  this  point,  you  should  separate 
the  critical  tasks  into  groups  of  systems  so  that  one  matrix  may  be  con- 
structed for  each  group. 

NO.  If  this  answer  is  chosen,  proceed  directly  to  Step 

List  all  critical  tasks  across  top  of  matrix.  Maintain  the  original 
wording  of  the  task  as  it  appears  in  the  soldier's  manual  when  filling  out  the 
top  portion  of  the  matrix.  This  helps  to  eliminate  confusion  later.  It  also 
helps  because  the  objective  of  this  matrix  is  n^  to  redefine  task  statements. 
By  this  time  you  should  be  able  to  consider  the  task  statement  as  valid.  All 
you  are  required  to  do  at  this  point  is  list  the  tasks  selected  in  Step 
across  the  top  of  the  matrix. 

Select  behavior  designators  and  list  them  vertically  to  form  the 
left-hand  column  of  the  matrix.  Behavior  designators  are  those  verbs  which 
denote  specific  skills  or  knowledge  necessary  for  task  element  accomplishment. 
Examples  are  shown  in  the  left-hand  column  of  the  sample  matrix  (Figure 
A-1).  The  selection  of  behavior  designators  is  accomplished  through  a 
search  of  the  JTSS  (Table  A-1)  Task  Data  Cards  (TDC),  or  other  similar 
documentation  which  details  the  actual  performance  steps,  or  task  elements, 
for  each  critical  task. 

While  the  concept  of  this  matrix  is  applicable  to  the  whole  field  of 
MOSs,  the  behavior  designators  selected  will  be  considered  as  unique  to  the 
set  of  tasks  being  analyzed.  This  is  because  the  same  verb  may  be  used  to 
designate  different  behaviors  in  different  MOSs.  For  example,  the  verb  "oil," 
when  used  in  describing  an  element  of  the  task  "Oil/Wet  concrete  fonns," 


denotes  a quite  different  action  than  when  used  in  a vehicular  maintenance 
task.  At  times,  the  same  verb  will  be  used  to  describe  different  behaviors 


within  the  same  MOS. 


For  example,  the  verb  "saw"  describes  one  skill  when  used  in  reference 
to  wood  and  another  when  used  in  reference  to  concrete.  In  this  situation, 
the  analyst  would  simply  include  the  modifier  as  part  of  the  designator, 
such  as  saw  (wood)  or  saw  (concrete).  For  example,  the  task,  "Cut 
and  install  batter  boards"  appears  on  the  JTSS  as  shown  in  Table  A- 1. 
The  appropriate  designators  are  underlined. 

Remember,  you  are  looking  for  VERBS,  words  that  describe  actions, 
something  the  soldier  must  ^ in  order  to  accomplish  the  task.  Many 
times  the  same  verb  will  be  used  in  each  of  a dozen  performance  steps 
of  a single  task.  An  example  of  this  is  found  in  the  task  "Identify 
construction  material  by  type  and  size."  That's  fine;  the  verb  "identify" 
is  describing  basically  the  same  action  each  time.  Whether  the  soldier 
must  identify  nails  or  grades  of  lumber,  it  is  still  basically  the 
same  action.  Simply  write  the  word  "identify"  in  the  left-hand  column 
and  go  on  to  the  next  desigfnator  or  to  the  next  JTSS  if  there  are 
no  more  different  designators  in  that  task. 

Analyze  the  JTSS  for  each  task  and  plot  the  designators  by 
checking  them  off  on  the  matrix  as  they  apply  to  each  task.  You  should 
begin  with  the  JTSS  for  the  first  task  you  have  listed  at  the  top  of  the 
matrix  as  has  been  done  in  Figure  A-1. 

Each  behavior  designator  is  gpven  equal  weight.  Thus,  only  one 
check  (or  point)  would  be  given  per  identified  behavior  per  task 
so  that  although  a single  task  may  contain  many  performance  steps  where 
certain  behaviors  occur  more  times  than  others . none  would  be  weighted 
more  heavily  than  any  other. 

Notice  that  although  the  task  in  the  JTSS  lists  two  separate  cutting 
actions,  once  for  the  posts  and  once  again  for  • the  boards  themselves, 
the  designator  "cut"  would  receive  only  one  point . This  insures  that  each 
action  receives  the  same  point  value  or  "weight"  in  the  matrix. 

Sum  designators.  Once  the  matrix  is  complete,  that  is. 
after  all  critical  tasks  have  been  accounted  for.  sum  the  behavior 
designators  horizontally  across  each  task.  This  step  establish«'s 


TABLE  A 1.  JOB  TASK  SUMMARY  SHEET 


Task;  Cut  and  install  batter  boards  Task  Criticality  (Circle  One)  I N * 

Materials,  Tools, 

Steps  in  Performance  Standard  of  Performance  Equipment 


1.  Cut  12  batter  board 
posts  and  sharpen  one 
end 


2.  Emplace  batter  board 
post  at  corners 


Posts  will  be  cut  long  enough 
so  that  when  ^FTven  firmly 
into  the  ground  the  posts 
will  extend  above  required 
finish  elevation  of  the 
foundations  as  directed  by 
crew  chief. 

3 batter  board  posts  will  be 
firmly  driven  into  the  ground 
3 or  4 feet  outside  of  each 
corner  post  as  directed  by 
crew  chief. 


2x4  material,  6 ft 
folding  rule,  square, 
crosscut  saw, 
half  hatchet 


12  2x4  stakes 

maul  or  sledge, 
folding  rule, 
framing  square 


3.  Measure  and  cut  batter 
board 


Batter  boards  will  be  cut 
long  enough  to  be  securely 
fastened  from  center  post  to 
outside  post  as  directed  by 
crew  chief. 


1 X 6 material, 
folding  rule, 
square,  crosscut 
saw 


4.  Attach  batter  boards  to 
posts 


Batter  boards  will  be 
securely  nailed  to  the  posts, 
level  and  at  exact  elevation 
of  finish  foundation  as 
directed  by  crew  chief. 


1 x 6 batter  boards, 
claw  hammer, 
folding  rule, 
carpenter's  level, 

8d  common  nails 


The  behavior  designators  are:  Cut,  Sharpen,  Emplace,  Measure  and  Nail** 

* Task  Criticality  Code 

C Critical 
I Important 
N Not  important 

**  The  designator  "nail"  is  used  here  instead  of  "attach"  as  "nail"  appeared  to  be 
the  more  definitive  designator.  "Attach"  would  probably  also  be  in  your  list. 

The  important  thing  to  remember  is  that  only  one  or  the  other  would  be  checked 
as  checking  both  designators  for  the  same  action  would  result  in  an  improper 
weighting  of  the  action. 


(W 
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which  designators  occur  with  the  greatest  frciiuetuy  across  the  critical  tasks 
and  is  the  first  step  towards  identifying  commonality 

(l^  Establish  mean.  The  column  formed  by  these  totals  (Step  ) 

is  then  summed  vertically,  and  the  total  is  shown  in  the  lower  right-hand 
corner  of  Figure  A-1.  This  total  is  then  divided  by  the  number  of  values  (or 
entries)  in  the  column  to  establish  a mean.  This  gives  you  the  average 
number  of  tasks  in  which  a designator  occurs. 

Having  established  a mean  number  of  tasks  in  which  a behavior 
occurs,  those  behaviors  which  occur  across  tasks  with  a frequency  at  or 
above  the  mean  are  considered  common.  You  now  go  through  the  matrix  and 
circle  the  check  marks  of  every  element  that  is  common . 

(T^  Evaluate  designators.  At  this  point  you  look  for  behaviors 
which  are  critical  even  though  they  may  not  be  common  For  example,  the 
designator  "vibrate  (concrete)"  In  Figure  A-1  is  not  identified  as  common. 
You  as  a subject  matter  expert,  however,  may  consider  it  to  be  a behavior 
which  is  essential  to  mastery  at  this  skill  level.  You  would  therefore  circle 
the  check  marks  applying  to  that  designator  so  that  tasks  which  incorporate 
it  are  given  an  extra  "weight"  which  will  result  in  the  task  being  ranked 
higher  in  Step  (l^  Remember,  the  matrix  is  a tool  to  aid  in  task 

selection  and  test  development;  as  such,  it  should  not  become  an  absolute 
basis  for  the  selection/rejection  of  test  item  candidates.  The  following 
criteria  are  given  as  a guide  to  evaluating  behavioi-  elements  for  importance 
and  criticality. 

(1)  The  degree  of  skill  required  in  the  use  of  tools,  equipment,  or 
communication  - the  higher  the  degree,  the  more  critical  the 
element . 

(2)  The  time  required  to  master  the  skill  - the  more  time,  the  more 
critical . 

Frequency  of  performance  of  the  skill  - the  more  frequent,  the 
more  critical. 

Consequences  of  failure  to  perform  - jeopardy  to  life  and  equipment 
equals  criticality . 

(5)  Degree  and  caliber  of  reaction  required  - unfailing,  rapid 
performance  under  all  conditions  equals  criticality. 


(3) 

(4) 


Vertically  sum  all  circled  (common  and/or  critical)  designators 
under  each  task.  This  enables  you  to  establish  which  tasks  are  the  prime 
candidates  for  inclusion  in  an  SQT. 

Rank  order  tasks.  With  common  behavioral  elements  preliminarily 
identified,  preliminary  candidate  tasks  for  inclusion  in  the  SQT  are 
rank  ordered  according  to  the  number  of  separate  common  behavioral 
elements  each  contains.  Thus,  the  task  or  tasks  with  the  greatest 
number  of  circled  check  marks  would  become  the  first  task  selected 
for  scanning  in  the  Simulation  Procedures. 


NOTES  1 ( ) numbers  indicate  the  sum  of  those  designators  which  occur  more 

than  the  established  mean  (2.41  & are  further  identified  by  (x^  . 
Detailed  explanation  contained  in  Step  of  the  text. 

2 Numbers  circled  e g.,  correspond  to  the  appropriate  step  in  the 

algorithm  & are  explained  in  detail  in  the  text  of  the  handbook. 


r 
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Total  number  of  behavioral  designators. 


2.4 


X 


Average  number  of 
behavior  designator 


tasks  in  which  a 
appears. 


Rank  order  of  the  tasks  based  on  the 
neatest  number  of  circled  checkmarks. 


Figure  A-1.  Task  Selection 
Matrix 


Select  all  beha.Jwf  d'^ignators  and  list  ti.em 
verKca'l.  to  form  left  hand  column  of  matrix. 


Beginning  with  job  tasl-  summary  sheet 
(JTSS)  Of  task  data  cards  <TDC)  of 
first  task  in  matrix,  check  oi'  applicable 
behavior  designators  Con'  nje  until 
all  tasks  nave  been  accourited  for 


Sum  the  checkmarks  hofuontally  for 
each  designator  to  form  right  hand 
column  of  matrix 


Sum  right-hand  column  of  matrix 
and  divide  answer  by  total  num- 
ber of  values  to  establish  mean. 


Circle  each  checkmark  of  each 
designator  whose  total  falls  at 
or  above  the  mean  to  identify 
common  behaviors. 


Evaluate  designators  for  impor- 
tance and  criticality. 


Vertically  sum  all  circled 
(common)  designators  under 
each  task. 


Rank  order  all  tasks  by  degree 
of  commonality 


! 

i 

1 

1 

Figure  A-2.  Task  Selection  | 
Procedures 
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APPENDIX  B 


SIMULATION  PROCEDURES 


INTRODUCTION 


The  procedures  are  presented  as  an  algorithm  (see  Figure  B-1)  made  up 
of  a series  of  actions,  (questions,  answers,  and  decisions  involved  in  the 
development  of  the  audio-visual  perfonnance  test . Segments  of  the  algorithm 
are  displayed  at  the  end  of  each  section  for  easy  reference.  Actions  are  indi- 
cated by  a circle, (^;  questions  by  a diamond, answers  by  a square]  |,  ; 
and  discussions  are  enclosed  within  a rectangle,  The  procedures  are 

easy  to  follow.  It  has  .ST  steps,  and  an  explanation  is  providetl  for  each 
step.  In  Steps  1 through  Mi  actual  task  element  data  is  considered  to  pro- 
vide a partial  demonstration  of  how  the  proc«*dures  are  used. 

PRELIMINARY  TEST  MODE  SELECTION 

GENERAL 


The  tasks  selected  for  testing  must  be  analyzeti  to  determine  whether  or 
not  a realistic,  reliable,  and  valid  scoreable  unit  can  be  piesented  in  an 
audio-visual  (A/V)  test  mode.  Your  earlier  analysis  in  the  Task  Selection 
Procedures  provided  a rank  ordered  list  of  tasks  to  be  considered  in  de- 
veloping the  SQT.  In  selecting  the  mode  by  which  the  tasks  may  be  tested, 
one  test  mode  may  be  more  appropriate  than  another  for  a specific  task.  A 
hands-on  test  may  be  the  most  appropriati-  for  one  task,  while  a written  or 
A/V  mode  would  be  appropriate  for  anothi'r 

This  section  of  the  procedures  is  primarily  designed  to  aid  in  making  a 
preliminary  determination  of  whether  a task  is  suitable  for  testing  by  A/V 
However,  in  th<'  proi'ess  it  bei'omes  neci'ssai'v  to  iilentify  whether  a perform- 
ance or  writli'n  ti'st  is  appropriate. 


rHKCKUlNG  FAOli  hlJlNK 
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Figure  B-2  depicts  Steps  1 through  11  and  shows  the  sequence  of  oper- 
ations involved  in  making  a preliminary  selection  of  test  mode. 

SIMULATION  ALGORITHM 

Identify  the  critical  elements.  Each  task  being  considered  for 
audiotvisual  simulation  testing  has  been  identified  as  a candidate  task  to  be 
covered  within  the  SQT.  The  purpose  of  this  first  step  is  to  identify  the 
parts  of  the  task  which  need  to  be  tested.  Each  task  includes  a number  of 
steps,  or  elements,  which  are  listed  on  the  JTSS/TDCs  or  in  the  soldier's 
manual.  Some  of  the  elements  must  be  accomplished  with  a high  degree  of 
accuracy  if  the  task  is  to  be  finished  in  an  acceptable  manner;  other  elements 
must  be  accomplished,  but  some  error  can  be  tolerated  without  serious  effect 
on  the  performance  quality  of  the  task.  The  elements  that  must  be  accom- 
plished with  a high  degree  of  accuracy  are  the  critical  or  key  elements,  and 
therefore  should  be  selected  for  testing.  For  some  tasks  all  elements  may  be 
critical,  but  for  others,  all  may  not. 

The  determination  of  critical  elements  must  be  made  by  personnel  who 
are  skilled  and  experienced  in  the  task  (i.e,,  subject  matter  experts). 
Ideally,  at  least  three  subject  matter  experts  should  be  involved  in  this 
process.  The  end  product  of  this  step  is  a list  of  the  critical  elements  asso- 
ciated with  a given  task. 

EXAMPLE  - Throughout  this  explanation,  we  wiU  use  the  MOS  51B  task, 
"Direct/Control  Placing  and  Finishing  Concrete"  as  our  example.  It  will  be 
assumed  that  all  elements  of  this  task  are  critical.  These  are  listed  in  Col- 
umn 1 of  Table  B-1. 

Analyze  each  task  element  regarding  its  perceptual,  action,  and 
decision  components.  This  step  will  help  you  determine  the  best  test  mode 
for  each  of  the  critical  element.s  listed  in  the  element  analysis  table  by  as- 
sessing the  importance  of  three  component  activities  in  each  element.  The 
three  component  activities  are  perceptual,  action,  and  decision. 

Perceptual  - That  component  of  a task  element  which  involve^ 

judgments  based  upon  the  senses  (see,  hear,  touch, 
taste,  smell). 


72 


iHiaatSSi 


Devek  p c<  »erion  fefeffn:ed  «icofjng 
procedures  ir*  dt.urdaiict  the 

Mjnja!  for  Dwelopir-q  SOT’S 


Review  tests  witn  five  subject  matter 
experts 


Review  test  script,  format  and 
scoring  with  five  other  subject 
matter  experts 


Is  the  A/V  presentation  of  acceptable 
quality^ 


Is  the  test  judged  technicaMy  sourid^ 


Is  the  test  of  adequate  reliability’ 


Develop  A/V  simulation  test  and 
response  forms 


Is  the  test  of  adequate  validity’ 


Figure  B-1.  Simulation  Algorithm 
(Sheet  2 of  2) 


51 

No 

50 
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3.  Direct/Control  placing 
concrete  into  wall, 
beams,  and  girder 
forms 

4.  Direct/Control  use  of 
vibrator 

5.  Direct/Control  screed- 
ing  of  concrete 

6.  Direct/Control  finish- 
ing concrete  using  a 
wood  float 

7.  Use  long-handle  wood 
float 

8.  Direct/Control 
finishing  concrete 
using  steel 
finishing  trowel 
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Action  - That  component  of  a task  element  involving  bodily 

movement  (motor  skills). 

Decision  - That  part  of  a task  element  which  involves  using 

past  knowledge  and  new  information  to  determine 
when  or  how  to  perform  the  task  element. 

Most  all  tasks  contain  perceptual,  action,  and  decision  components.  To 
identify  these  components  in  a task  element,  a subject  matter  expert  should 
consider  the  following  three  questions: 

(1)  When  errors  occur  in  the  performance  of  this  task  element,  is  it 
because  people  fail  to  perceive  (see,  hear,  feel,  taste,  smell) 
important  information? 

If  the  answer  to  the  question  is  "YES,"  the  task  element  has  a 

critical  perceptual  component. 

(2)  When  errors  occur  in  the  performance  of  this  task  element,  is  it 
because  people  fail  to  make  coordinated  or  pi^ecise  bodily  move- 
ments? 

If  the  answer  to  the  question  is  "YES,"  the  task  element  has  a 

critical  action  component. 

(3)  When  errors  occur  in  the  performance  of  this  task  element  is  it 

because  people  are  misusing  knowledge  or  information? 

If  the  answer  to  the  question  is  "YES,"  the  task  element  has  a 

critical  decision  component. 

EXAMPLE  - The  critical  components  for  the  elements  of  the  task  to 
"Direct/Control  Placing  and  Finishing  Concrete"  are  listed  in  Column  2 of 
Table  B-2.  You  should  note  that; 

(1)  Only  one  task  shows  a critical  action  component,  all  the  other 
elements  require  directing  others  rather  than  doing  the  act. 

(2)  The  first  three  elements  contain  only  critical  decision  components 
because;  (a)  it  is  considered  likely  that  errors  in  those  elements 
would  result  from  failure  to  give  proper  direction  even  though  one 
perceived  the  situation  accurately,  and  (b)  the  elements  do  not 
involve  the  actual  placing  of  ramps  or  concrete. 


TABLE  B-2.  ELEMENT  ANALYSIS  (B) 

Example  Analysis  of  Critical  Elements  for  Task 
"Direct/Control  Placing  and  Finishing  (Concrete” 


Critical  Elements 


Critical  Components 


Stimulus  Variables 


Column  1 


Column  2 


Column  3 


1.  Direct/Control  placing  of 
ramps 

2.  Direct/Control  placing  of 
concrete  for  slab 
construction  or  small  paved 
surfaces  on  grade 

3.  Direct/Control  placing 
concrete  into  wall, 
beams,  and  girder  forms 

4.  Direct/Control  use  of 
vibrator 

5.  Direct/Control  screeding 
of  concrete 

6.  Direct/Control  finishing 
concrete  using  a wood 
float 

7.  Use  long-handle  wood 
float 

8.  Direct/Control  finishing 
concrete  using  steel 
finishing  trowel 


Decision 


Decision 


Decision 


Perceptual/Decision 


Perceptual/Decision 


Perceptual/Decision 


Action/Perceptual 


Perceptual/Decision 


(3)  Elements  4,  5,  6,  and  8 contain  critical  perceptual  and  decision 
components  because  errors  are  more  likely  to  occur  when  either: 
(a)  the  person  fails  to  recognize  the  consistency,  wetness,  or  level 
of  the  concrete  relative  to  the  operations  that  must  be  performed , 
or  (b)  the  person  recognizes  the  consistency,  wetness,  or  level  but 
directs  an  operation  to  begin  or  end  at  the  wrong  time. 

Do  all  critical  elements  present  only  action  components?  This  ques- 
tion is  asked  to  determine  whether  the  task  should  be  tested  by  a perform- 
ance test  or  considered  for  testing  via  written  or  audio-visual  simulation 
mode.  This  question  is  easily  answered  by  referring  to  your  element  analysis 
table . 

|T|  YES.  If  this  answer  is  selected,  it  leads  to  the  recommendation 
that  a performance  test  be  developed  for  the  task.  The  assumption  is  that 
the  action  component  of  a task  element  is  best  measured  via  a performance 
test,  and  that  when  all  elements  present  only  critical  action  components,  a 
performance  test  is  especially  justified. 

|T|  NO.  If  this  answer  is  selected,  it  is  necessary  to  gain  more  infor- 
mation, which  is  done  by  moving  on  to  Step 

EXAMPLE  - Column  2 of  Table  B-2  shows  that  there  were  critical  decision  or 
perceptual  components  for  at  least  one  task  element.  Therefore,  the  answer 
to  the  question  presented  in  Step  is  "NO."  A "YES"  answer  would 

require  that  each  task  element  has  only  a critical  action  component . 

Do  all  critical  elements  present  only  critical  decision  components? 
The  answer  to  this  question  determines  whether  the  task  should  be  tested  by 
a written  test  or  if  an  audio-visual  test  format  can  be  used.  The  question  is 
easily  answered  by  referring  to  your  element  analysis  table. 

|T|  YES.  If  this  answer  is  selected,  it  leads  to  the  recommendation 
that  a written  test  be  developed  for  the  task.  The  assumption  is  that  when 
the  decision  component  is  the  only  critical  component,  it  can  be  measured 
most  economically  via  a written  test. 

8 NO.  If  this  answer  is  selected,  it  is  necessary  to  gain  more  infor- 
mation which  is  done  by  moving  on  to  Step 
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EXAMPLE  - Table  B-2  shows  that  there  were  critical  decision  or  perceptual 
components  for  at  least  one  element.  Therefore,  the  answer  to  Step 
is  "NO."  If  each  task  element  had  onj^  a critical  decision  component,  the 
answer  would  have  been  "YES." 

Do  all  critical  elements  present  only  critical  action  and  decision 
components?  This  question  is  asked  to  deteniiine  whether  a written  and/or 
performance  test  should  be  used  or  if  an  audio-visual  foniiat  should  be 
considered.  This  question  is  easily  answered  by  referring  to  the  element 
analysis  table. 

Y'ES.  If  this  answer  is  selected,  it  is  recommended  that  a written 
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and/or  performance  test  be  developed  for  the  task.  Typically,  a performance 
test  is  preferred.  However,  sometimes  the  decision  component  of  a task  or 
subtask  can  be  tested  adequately  via  performance  testing  only  by  many 
repetitions  of  the  action  components.  This  can  be  impractical.  For  example, 
a simple  performance  test  of  one's  ability  to  construct  a given  type  of  roof 
may  adequately  measure  the  action  components  (measure,  saw,  nail")  common  to 
many  types  of  roofs,  but  it  may  not  adequately  test  the  decision  components 
(selection  of  materials,  determining  sequence  of  events')  that  vary  according 
to  the  type  of  roof.  In  this  situation  a written  test,  which  addresses  the 
decision  components,  might  be  used  along  with  the  performance  test  which 
focuses  on  the  action  components . 

NO.  If  this  answer  is  selected,  then  by  process  of  elimination,  you 
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have  isolated  those  task  elements  which  may  feasibly  be  tested  in  an  A/V 
mode.  It  is  now  necessary  to  gain  more  infonnation  which  is  done  by  moving 
to  Step  (1^ 

EXAMPLE  - Column  2 of  Table  B-2  shows  that  there  are  perceptual  compo- 
nents as  well  as  action  and  decision  components.  Therefore,  the  answer  to 
this  step  is  "NO."  If  only  action  and  decision  components  were  listed,  the 
answer  would  have  been  "YES." 

TEST  REALISM 


GENERAL 


Since  you  have  detemined  that  a task/element  has  perceptual  content 
and  is  a candidate  for  A/V  testing,  further  analysis  becomes  necessary.  To 


Identify  the  critical  elements 


Analyze  each  critical  element  regarding 
Its  perceptual,  action  and  decision 
components 


Do  all  critical  elements  present  only 
critical  action  components? 


Develop  a performance  test  in 
accordance  with  TRADOC  Doctrine 


Do  all  critical  elements  present 
only  critical  decision  components^ 


Develop  written  test  in  accordance 
with  the  TRADOC  Doctrine 


Do  all  critical  elennents  present  only 
critical  action  arid  decision  com 
ponents? 


Develop  performance  arui/or  written 
test  in  accordance  with  TRADOC 
Doctrine 


Figure  B-2.  Preliminary  Test  Mode  Selection 


provide  a valid  test,  the  A/V  mode  must  present  the  task/element  in  a real- 
istic manner.  Figure  B-3  (Steps  12  through  17),  at  the  end  of  this  section, 
outlines  the  sequence  of  operations  leading  to  this  determination. 

SIMULATION  ALGORITHM 

(l^  Identify  and  list  the  stimulus  variable  associated  with  each  critical 
element.  The  purpose  of  this  step  is  to  analyze  each  task  element  and  specif- 
ically determine  what  a person  responds  to  as  he  performs  the  task  element. 
For  example,  in  deciding  when  to  use  a wood  float  for  finishing  concrete,  it 
is  not  adequate  that  a person  respond  only  to  the  "appearance"  of  the  con- 
crete or  to  the  amount  of  moisture  in  the  concrete.  There  are  certain  clues, 
or  stimulus  variables  that  permit  a person  to  judge  the  appearance  or  mois- 
ture of  the  concrete,  and  each  of  these  stimulus  variables  must  be  listed. 

EXAMPLE  - In  Column  3 of  Table  B-3,  five  possible  stimulus  variables  have 
been  listed  for  the  sixth  critical  step  - "Direct/Control  finishing  concrete 
using  a wood  float."  This  example  indicates  that  in  directing  the  finishing  of 
concrete  using  a wood  float,  a person  responds  to  certain  characteristics  of 
the  appearance  of  the  concrete;  specifically , he  determines  when  to  start  and 
stop  based  upon  the  five  stimulus  variables  listed  in  Column  3. 


Determine  the  importance  of  each  stimulus  variable  within  each 
critical  element.  The  purpose  of  this  step  is  to  further  define  the  stimulation 
requirements  of  a given  task  or  task  element.  In  performing  this  step, 
subject  matter  experts  should  refer  to  the  stimulus  variables  which  were 
identified  in  Step  and  evaluate  their  importance  to  the  proper  per- 

formance of  the  critical  element.  Simulation  usually  degrades  some  aspects  of 
the  stimulus.  If  this  step  is  performed  properly,  it  helps  anticipate  the 
effect  of  any  degradation. 

Importance  is  difficult  to  determine  because  many  factors  go  into  making 
something  important.  In  this  step  a procedure  is  suggested;  but  it  is  recog- 
nized that  the  subject  matter  experts  will  have  to  be  subjective.  The  subject 
matter  expert  should  answer  the  following  question  in  determining  the  impor- 
tance of  a stimulus  variable.  "Considering  all  of  the  information  which  is  pro- 
vided by  all  of  the  stimulus  variables  which  are  typically  present  when  this 
task  element  is  performed,  how  often  does  this  one  stimulus  variable  provide 
unique  information  that  is  essential  to  proper  performance  of  the  task  element?" 
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TABLE  B-3.  ELEMENT  ANALYSIS  (C) 


Example  Analysis  of  Critical  Elements  for  Task: 
"Direct/Control  Placing  and  Finishing  Concrete" 


Critical  Element 

Critical  Components 

Stimulus  Variables 

Column  1 

Column  2 

Column  3 

1. 

Direct/Control  placing  of 
ramps 

Decision 

2. 

Direct/Control  placing  of 
concrete  for  slab  con- 
struction or  small  paved 
surface  on  grade 

Decision 

3. 

Direct/Control  placing 
concrete  into  wall,  beams, 
and  girder  forms 

Decision 

4. 

Direct/Control  use  of 
vibrator 

Perceptual/Decision 

5. 

Direct/Control  screeding 
of  concrete  using  a 
wood  float 

Perceptual/Decision 

6. 

Direct/Control  finishing 

Perceptual/Decision 

a. 

Uniformity  of  color 

concrete  using  a wood 

of  concrete 

float 

b. 

Presence  or  absence  of 
swirls  in  concrete 

c. 

Presence  or  absence  of 
pits  in  concrete 

d. 

Presence  or  absence  of 
pockets  of  water 

e. 

Firmness  of  concrete 
in  response  to  slight 
pressure 

7. 

Use  long  handle  wood 
float 

Action/Perceptual 

8. 

Direct/Control  finishing 

Perceptual/Decision 

concrete  using  steel 
finishing  trowel 


Obviously,  a stimulus  variable  is  very  important  if  it  always  pi’oviiles 
unique  information  (i.e.,  information  that  is  not  available  from  any  olhei' 
stimulus  variable)  which  is  also  essential  to  proper  performance.  A stimuhis 
characteristic  will  be  much  less  impoi'tant  if  it  either  duplicates  information 
already  available  or  if  its  informational  value  has  a less  direct  effect  upon 
proper  perfonaance . Kach  stimulus  variable  should  be  labeled  as  very 
imporbjnl,  moderately  important,  or  not  very  important. 

KXAMPLK  - In  performing  this  step,  the  subject  matter  experts  will  review 
the  stimulus  variables  .associatoil  with  a task  element  See  t\)lumn  d of  Table 
U-3  and  note  the  five  stimulus  variables  associated  with  th<'  sixth  critical 
element.  A juiigment  is  now  made  of  the  impoiMance  of  each  stimulus  variable. 
The  subject  matter  expert  should  follow  the  procetlui’e  shown  in  Tabh'  M-  J 
In  that  table,  each  stimulus  variable  is  rated  as  providing  either  uniqu^  or 
non-unique  information,  and  as  providing  information  which  is  essential  or 
non-essential  to  the  outcome  of  the  critical  element  . 

In  the  example  prov’ided  in  Table  B-4.  a stimulus  variable  is  rateil.  (1) 
very  important  if  it  was  judged  both  unitiue  and  essential,  and  (2)  moderately 
important  if  it  is  either  uni<iue  or  essential,  but  not  both.  A stimulus  vari- 
able would  be  rateil  as  not  very  important  if  it  is  neither  uniiiue  nor 
essential . 

(i^  Make  a preliminary  determination  of  the  realism  of  stimulus  presen- 
tation on  paper,  e.g..  ilrawing  or  picture  and  via  intended  A/V  simulation 
device,  e.g..  'TV.  slid<;/tape.  'The  purpose  of  this  sfej>  is  ti^  start  determin- 
ing whether  a critical  element  is  best  tested  via  an  A/V  device  or  a paper- 
and-pencil  format.  Both  the  subject  matter  expert  and  a training  media 
expert  must  work  together.  They  will  look  at  ea<’h  stimulus  v.ariabli'  s«'lected 
>n  Step  (1^  and  determine  which  will  be  the  most  realistic  method  of  presen- 
t.itum;  (1)  a drawing  or  picture  on  .ainlio-visual  dc'vict' 

(eg,  TV,  slide/tape).  In  many  cases  there  will  be  litth*  or  no  dtffcrroct' . 
t<ut  there  are  at  least  three  situations  in  which  A/V  can  add  to  the  realism  of 
I .iimulus  presentation.  First.  A/V  is  advantageous  when  the  observation  of 
1.  li.-n  'I-  of  constantly  changing  physical  characteristics  of  an  environim’nt  is 
• t ml  Second,  three-dimensional  relationships  can  be  piesi'uti'd  nu're 
. VI » A V Finally,  the  coorilinatum  of  st'uinl  and  visual  stimuli  is 
I . U t.  Complished  vi.i  A V. 
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TABLE  B4.  DETERMINING  THE  IMPORTANCE  OF 
EACH  STIMULUS  VARIABLE  ASSOCIATED  WITH  A CRITICAL  ELEMENT 


i 

3 

} 


i 

] 


( 


i 

.t 


Ciittcal  Element:  Direct/Control  Finishing  Concrete  Using  a Wood  Float 


Slinntlirs  Variables 

Unique  * 

Essential  * 

lm()oitance  * 

Umtormity  ot  color  ot  corrcretr: 

Yes 

No 

Modmalely  Important 

Presence  or  absence  ot  swirls 
in  concrete 

Yes 

Yes 

Very  Important 

Pre.serice  or  absence  ot  pits 
irt  concrete 

Yes 

Yes 

Very  Ingrortant 

Presence  or  absence  of  frockets 
ot  water 

Yes 

Yes 

Very  Impoitant 

Firnrness  ot  concrete  in  resfrorrse 

Yes 

Yes 

Very  Impoitant 

to  slight  ptessiite 


‘Ratings  arc  tor  purpose  ot  example  and  are  not 
trecessarily  valid  in  describing  tire  variable. 


T’ht're  ace  also  coinlitions  which  woulil  favor’  a di’awing:  or*  piotufo.  Small 
color’  iliffor'cnces  will  be  pr’r'sertted  mor'e  faithfirlly  with  photoj^jr’aphs  tharr  TV. 
especially  wherr  'TV  testiog  involves  pfesentatrons  over  nrany  T’V  sets  which 
are  in  varying  states  of  repair  atni  adjustment  Likewise  when  fine-line  defi- 
nitu>n  is  requirr'd,  the  ilrawing  or  photograph  will  ofterr  be  preferrt'd . 

EXAMPLE  - Five  stinuihrs  variables  front  I'olutntt  of  the  element  atutlysis 
table  are  listerl  again  in  the  left-hartd  colvirnn  of  Table  H-5.  An  estimate  is 
ni>w  marie  of  whether  a tlrawing  or  pictitr*e  versits  an  A/V  presentatiorr  of  the 
stimirlus  vat’iablr's  will  (rrovirle  grt'attM’  realism  In  this  srtuatirrn  th«'  A \’ 
tnr'dia  is  televisu>n.  so  thr'  realism  of  a leh'viseil  presmitatirtn  is  consith>r’»'rl 
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TABLE  B-5.  DETERMINING  THE  APPROPRIATENESS  OF  AN  AUDIO- 
VISUAL PRESENTATION  MODE  FOR  A SIMULATED  SKILL  OUALIFICATION  TEST 

Critical  Element;  Direct/Control  Finishing  Concrete  Using  a Wood  Float 


(1) 

Stimulus 

Variables 

(2) 

Importance 

(3) 

Realism 

(4) 

Recommended 
Presentation  Format 

Uniformity  of 

Moderately 

•Still  picture 

Still  picture 

color  of  concrete 

important 

acceptable  - TV 
questionable 

(if  tested) 

Presence  or 
absence  of  swirls 
in  concrete 

Very  important 

TV  acceptable 

Still  picture 
acceptable 

TV 

Presence  or 
absence  of  pits 
in  concrete 

Very  important 

TV  acceptable 

Still  picture 
acceptable 

TV 

Presence  or 
absence  of 
pockets  of  water 

Very  important 

TV  acceptable 

Still  picture 
acceptable 

TV 

Firmness  of 
concrete  in 
response  to 
slight  pressure 

Very  important 

TV  acceptable 

Still  picture 
acceptable 

TV 

•Preferred  presentation  format 


Sometimes  it  will  be  necessary  to  produce  a test  item  in  two  or  more  for- 
mats to  determine  the  most  realistic  format,  but  at  this  time  an  estimate  is 
made  based  on  previous  experience.  Estimates  of  the  most  acceptable  method 
of  presentation  for  each  of  the  stimulus  variables  are  listed  in  the  fourth 
column  of  Table  B-5.  The  reasons  for  these  estimates  are; 

(1)  A still  picture  is  preferred  for  the  first  stimulus  variable  because  it 
is  anticipated  that  it  will  be  difficult  to  maintain  a constant  presen- 
tation of  small  color  differences  of  concrete  when  shown  on  different 
TV  sets. 
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(.2)  A televised  presentation  is  preferred  for  the  second , thiid,  aiul 
fourth  stimulus  vai’iables  because  these  variables  are  typically 
observed  as  one  scans  and  moves  around  the  perimeter  of  the 
concrete.  Television  will  provide  for  greater  realism  in  simulating  the 
behavior  which  occurs  as  these  stimulus  variables  are  observed. 

Is  it  probable  that  the  A/V  simulation  will  prov'itit*  acceptable  realism 
in  presenting  very  important  stimulus  variables?  One  may  wonder'  why  fourteen 
steps  precede  such  an  obvious  question.  The  I'ationale  up  to  this  point  is  that: 
(1)  the  pt'esence  of  perceptual  content  should  be  established  before'  conside'ring 
the  use  of  A/V,  and  (2)  the  specific  simulation  r-eciuii'cments . whie'h  ai-e  the 
stimulus  variables,  must  be  identified  before  rejecting  or  accepting  an  A/V 
mode.  'This  step  is  to  confinn  a preliminai'y  determination  of  whe'tlu'r  or  ni>t  to 
use  A/V.  'The  answer  to  this  step  is  based  upon  the  entries  in  ('olumns  2 and  2 
of  Table  B-5.  Column  2 contains  the  judgments  made  earlier  in  Stej)  anil 

Column  3 contains  the  judgments  made  in  Step  (l^ 

[T^  YES.  This  answer  indicates  the  potential  use  of  an  A/V  format 
However,  further  analysis  is  requii'ed. 

[T^  NO.  This  answer  indicates  that  none  of  the  im|>oi'tant  stimulus 
variables  can  be  presented  by  A/V  with  act'«'{)table  r'e.rlisni  Consequt'ntly , 
pei'formance  or  written  testing  is  necessary 


EXAMPLE 


'The  entries  in  Columns  2 and  3 of  Table  B-,')  indicate  that  four  of 


the  five  stimulus  variables  judged  to  be  very  important  and  TV  is  the  j^refeired 
format  for  each  of  these.  Consequently,  a "Y’ES"  answer  is  clearly  indicateil 
If  acceptable  realism  was  possible  on  only  one,  two,  oi-  three  of  the  variables,  a 
"YES"  answer  should  still  be  given.  In  a later  stej>  (Step  (2'^)  ),  an  A/V  test 

format  will  be  rejected  if  it  is  jiulged  to  be  too  limited  in  st’oi)e. 

RESPONSE  REALISM 

GENERAL 


'To  determiiK’  whether  or  not  th<*  A A’  j)resentation  will  j)i'ovi4le  valid  test 
results,  it  is  necessary  to  vleternune  wlu'ther  the  test  resj>ons('s  ami  ji'b 
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Figure  B-3.  Test  Realism 
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responses  are  adequately  similar.  Figure  B-4,  (.Steps  18  through  23)  shows 
the  sequence  of  analysis  necessary  to  make  this  determination 


SIMULATION  ALGORITHM 


Specify  the  correct  response  to  each  stimulus  variable  when  it  is 
observed  in  the  job  environment.  A test  presents  stimuli  and  requires  re- 
sponses. Stimulus  realism  is  important,  but  response  realism  must  also  be 
considered.  The  purpose  of  this  step  is  to  describe  what  a person  does  on 
the  job  in  response  to  the  stimulus  variable  or  a cluster  of  stimulus  variables. 
Subject  matter  experts  should  perform  this  step  by  listing  each  stimulus 
variable  or  group  of  variables  that  are  appropriate  for  TV  simulation  and 
identifying  the  proper  action  that  would  result  from  the  stimulus. 

EXAMPLE  - Continuing  with  the  critical  element.  "Direct/Control  Finishing 
Concrete  Using  a Wood  Float."  In  the  preceding  steps  four  stimulus  variables 
were  designated  as  appropriate  for  a televised  test.  These  are  listed  in 
Column  1 of  Table  B-6.  Responses  on  the  job  are  listed  in  Column  2.  Since 
each  variable  has  at  least  two  states  (e  g.,  presence  or  absence),  at  least 

two  different  responses  will  be  needed  if  it  is  decided  to  test  for  all  states. 

In  this  case,  it  is  not  necessary  to  test  for  all  states  of  each  variable,  be- 
cause on  the  job  one  responds  only  to  the  presence  of  a state  (i.e..  swirls, 

pits,  water  pockets)  and  not  the  absence. 


Specify  the  possible  types  of  responses  to  each  stimulus  variable  in 
the  test  setting  and  select  the  most  appropriate  one.  One  requirement  of  the 
A/V  simulation  test  is  that  it  should  be  administered  to  a group  of  soldiers 
who  are  assembled  in  one  place  at  one  time.  Additionally,  at  this  time,  the 
use  of  computer  response  teiminals  or  actual  equipment  is  not  considered. 
Consequently,  responses  will  be  recorded  on  paper,  and  the  range  of  types 
of  responses  is  limited.  However,  it  is  emphasized  that  there  are  alternatives 
to  the  standard  multiple-choice  response  fomat . and  some  of  these  alterna- 
tives will  definitely  be  preferred  in  certain  situations. 

The  purpose  of  this  step  is  to  select  the  most  appropriate,  or  job-like 
response  that  is  available  for  the  A/V  simulation  test  format.  The  subject  mat- 
ter expert  should  use  his  ingenuity  in  designing  or  selecting  the  type  of  test 
response.  Four  categories  of  test  responses  are  now  discussed  in  some  detail. 


TABLE  B-6.  COMPARISON  OF  RESPONSE  REQUIREMENTS  TO  STIMULUS  VARIABLE 
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Preferred  ivi)e  of  response  to  test  item. 

Recommended  presentation  format  for  entire  critical  element 


fi 


1 


i 

I 
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(1)  Multiple-choice  response  - This  is  a typical  test  item  response 
format  which  requires  the  test  taker  to  recognize  the  correct  alter- 
native when  it  is  presented  with  two  or  more  incorrect  distractors. 
While  this  is  a common  test  response,  it  is  quite  limited  in  many 
types  of  job  activities.  Three  important  characteristics  of  the 
multiple- choice  responses  are; 

(.a)  The  test  taker  must  simply  recognize  the  correct  response  from 
a limited  number  of  alternatives;  he  is  not  doing  anything 
other  than  recognizing  a correct  response. 

(b)  The  test  taker  is  typically  alerted  that  the  correct  response  is 
present  among  the  limited  number  of  alternatives. 

(c)  The  test  taker  is  responding  to  a small  number  of  alternatives 
which  are  all  present  at  the  same  time. 

In  view  of  these  characteristics,  the  multiple- choice  format  will  be  appro- 
priate when  used  to  test  a job  response  in  which;  (1)  the  man  on  the  job 
selects  the  correct  action  from  a small  number  of  obvious  possible  actions,  (2) 
the  man  on  the  .job  knows  in  advance  that  one  of  the  obvious  possible  actions 
is  correct,  and  (3)  the  obvious  possible  actions  are  all  possible  at  the  same 
point  in  time.  The  multiple- choice  format  is  appropriately  selected  for  exam- 
ple, to  measure  one's  knowledge  of  what  type  of  hammer  to  use  to  drive  in  a 
spike  (assuming  that  on  the  job;  (a)  the  right  type  of  hammer  is  there  to  be 
selected,  (b)  the  man  on  the  job  knows  that  one  of  the  hammers  is 
right,  and  (c)  all  of  the  hammers  can  be  readily  observed  at  the  same  time). 
The  multiple- choice  cannot  be  used  to  measure  one's  ability  to  use  the  hammer 
(e.g.,  note  the  difference  between  recognizing  a good  golf  swing  and  doing 
it)  or  to  recognize  an  acceptable  concrete  finish,  unless  comparator  concrete 
slabs  are  always  present  on  the  job. 

(2)  Alerted  two- alternative  response  - This  type  of  response  refers  to 
true-falsq,  go/no-go,  good-bad,  accept- reject,  type  judgments  when 
the  test ' taker  is  aware  that  one  of  the  two  responses  is  correct. 
The  basic  differences  between  this  response  and  the  multiple-choice 
response  is  that  there  are  only  two  possible  judgments  as  opposed 
to  from  three  to  five  in  multiple-choice  items.  The  discussion  con- 
cerning the  multiple-choice  response  is  also  relevant  to  this  type 
response.  This  format  is  appropriate  to  test  inspection  require- 
ments where  a given  end  product  is  accepted  or  rejected;  but  it  is 


I 
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not  preferred  to  measure  job  performance  in  which  a supervisor 
makes  go/no-go  decisions  during  some  procedure,  such  as  when  to 
stop  vibrating  concrete. 

(3)  Unalerted  identification  response  - This  type  of  response  varies 
from  the  multiple-choice  and  alerted  two-alternative  response  in  a 
very  important  respect.  The  test  format  for  this  type  of  response 
is  one  in  which  the  test  taker  observes  a sequence  of  events  (for 
example,  a construction  team  budding  the  frame  of  a building).  He 
is  instructed  to  record  whenever  he  identifies  certain  types  of 
events  (for  example,  violations  of  safety  precautions,  deviations 
from  construction  prints,  improper  use  of  tools).  One  way  to 
record  this  answer  on  a structured  answer  sheet  is  to  • perimpose 
a clock  on  the  visual  test  presentation,  and  the  test  taker  records 
the  time  of  his  identifying  response.  This  type  of  response  is  most 
appropriate  when  measuring  one's  ability  to  identify  critical  events 
as  they  occur  in  time. 

(4)  Unalerted  decision  response  - This  type  of  response  is  required 
when  a test  item  presents  a question  but  does  not  present  specified 
alternative  responses.  The  correct  response  might  be  anything 
from  a number  or  letter  symbol  to  a paragraph  of  writing.  This 
type  of  response  is  difficult  to  incorporate  into  a standardized 
objective  test,  but  with  some  ingenuity  it  can  be  done  for  specific 
applications.  For  example,  if  a construction  drawing  or  a picture 
of  a structure  were  included  as  answer  sheet,  some  answers  con- 
cerning interpretation  of  construction  drawings  could  be  marked  on 
the  drawing  or  the  picture.  This  type  of  response  more  closely 
approximates  the  typical  job  situation  in  which  one  must  correctly 
interpret  something  or  make  a decision  and  the  correct  response  is 
not  explicitly  provided  as  one  of  a number  of  alternatives. 

EXAMPLE  - Column  3 of  Table  B-6  (Step  (l^  ) presents  possible  and 

recommended  test  responses  to  each  of  the  four  stimulus  variables . These 
are  determined  by  considering  both  the  response  on  the  job  (Column  2 of 
Table  B-6),  and  the  types  of  test  responses  which  have  been  discussed  in 
the  preceding  paragraphs.  The  recommended  test  response  to  Stimulus  Vari- 
ables 1,  2,  and  3 in  Table  B-6  is  an  unalerted  identification  response,  which 


means  that  the  test  taker  should  be  required  to  identify  the  presence  of 
swirls,  pits,  or  pockets  of  water  if  they  occur  as  part  of  a test  item  that 
shows  the  placing  and  finishing  of  concrete.  The  test  taker  should  be  told  at 
the  onset  of  the  test  to  note  any  events  which  require  correction  as  they  are 
observed,  but  he  should  not  be  presented  a sequence  of  presentations  and 
asked  specifically  if  swirls,  pits,  or  pockets  of  water  are  present.  The  reason 
for  tiiis  is  that  on  the  job  the  supervisor  who  directs  the  placing  and  finish- 
ing of  concrete  must  likewise  detect  imperfections  as  they  occur  in  time;  he 
does  not  have  a discrete  set  of  alerted  times  in  which  to  look  for  a swirl  in  a 
given  area  or  pit  in  a given  area.  Further,  his  response  to  observing  the 
imperfection  is  to  report  his  observation  to  the  worker  who  corrects  it.  In 
the  test  situation  the  report  is  made  on  paper. 

Note  also  that  in  the  job  situation,  the  supervisor  does  not  respond  to 
the  absence  of  imperfections.  As  long  as  no  imperfections  are  observed, 
work  continues  without  input  from  the  supervisor.  Consequently,  it  is  not 
recommended  that  a test  taker  be  required  to  respond  to  a test  item  in  which 
he  reports  the  absence  of  imperfections. 

An  alerted  two-alternative  response  is  recommended  for  the  fourth 
stimulus  variable  because  that  is  the  type  of  response  on  the  job.  The 
supervisor  or  worker  intentionally  applies  pressure  as  a test  of  firmness,  and 
at  that  instant  the  supervisor  provides  a yes-no  response  as  to  whether  or 
not  floating  should  begin. 

Compare  the  on-the-job  versus  the  test  response  requirements  to 
each  stimulus  variable  and  evaluate  the  effects  of  differences.  The  purpose 
of  this  step  is  to  determine  whether  responses  on  the  job  and  on  a test  are 
sufficiently  similar  to  enable  a valid  test.  This  judgment  will  be  in  large  part 
subjective,  although  greater  objectivity  may  be  possible  after  experience  has 
been  gained.  Three  situations  in  which  response  dissimilarity  could  have  major 
negative  effects  on  test  validity  are  considered. 

(1)  Critic  versus  actor  response  - This  refers  to  a situation  in  which 
the  job  requires  a task  to  be  done  (e.g.,  vibrate  concrete)  and  the 
test  requires  one  to  observe  someone  else  do  the  job  and  evaluate 
good  or  bad  points . For  the  test  to  be  valid  it  is  necessary  to 
assume  that  recognizing  mistakes  is  the  same  as  not  making  mis- 
takes. One  should  be  hesitant  to  make  this  assumption  - in  sports 


or  the  perfonning  arts  it  is  obvious  that  critics  cannot  necessarily 

perform  although  they  are  proficient  at  recognizing  flaws.  This 

type  of  response  dissimiliarity  is  most  likely  to  arise  when  attempt- 
ing to  test  an  action  component  via  audio-visual  simulation , and  that 
is  a major  reason  why  Step  4 indicates  performance  tests 
should  be  used  if  only  action  components  are  involved. 

(2X  Level  of  distraction  of  job  versus  test  - Some  jobs  require  one  to 
attend  to  many  different  things  despite  a variety  of  demands  or 
interruptions.  For  example,  an  electronics  troubleshooter  may  have 
to  interrupt  his  troubleshooting  to  study  circuit  theory,  to  go  to  a 
parts  manual  or  to  go  pick  up  tools  or  test  equipment.  The 

switchboard  operator/receptionist  must  respond  to  many  calls  coining 
in  and  terminating  as  well  as  people  coming  and  going.  Any  test 
which  requires  a response  to  some  isolated  task  will  run  the  danger 
of  not  providing  valid  results  because  of  this  type  of  discrepancy 


between  the  job  environment  and  the  test  environment. 

(3)  Differences  in  difficulty  of  job  and  test  response  - This  situation  is 
similar  to  the  preceding  but  not  the  same.  In  this  case  both  the 
job  response  and  the  test  response  may  have  the  same  amount  of 
distraction  or  interruption , but  the  job  response  is  often  more 
difficult.  This  may  occur  because  there  really  are  a large  number 
of  job  responses  (e.g.,  a carpenter  may  hammer  many  types  of 
nails  into  many  types  of  wood  from  many  different  positions)  but  a 
test  must  generalize  from  only  one  or  a small  number  of  test 
responses  (e.g.,  hcimmering  one  nail  into  one  type  of  wood  while  in 
one  position).  Hammering  may  be  more  difficult  under  some  condi- 
tions than  others,  and  a test  which  selects  a less  difficult  condition 
may  be  of  limited  validity . 

Another  example  of  an  unrealistically  simple  test  response  can  occur  if 
the  test  alerts  the  test  taker  to  something  that  he  must  identify  while  on  the 
job  when  he  is  not  alerted.  This  is  why  Step  discusses  alerted  and 

unalerted  responses  in  some  detail. 

After  considering  the  differences  and  similarities  between  the  job  re- 
sponse and  test  response  it  is  necessary  to  decide  whether  the  responses  are 
adequately  similar  to  provide  valid  test  results.  This  judgment  will  be 
subjective. 
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EXAMPLE  - The  judgftnents  called  for  in  Step  (2^  are  listed  in  Column  4 
of  Table  B-6.  In  deciding;  that  each  of  the  four  preferred  test  responses  are 
adequately  similar  to  the  job  response  the  following;  points  are  considered 

(.1)  If  the  test  item  depicts  the  placing;  and  finishing;  of  concrete  from 
the  vantag;e  point  of  a supervisor,  the  test  taker  can  report  his  ob- 
servations of  defects  (swrirls,  pits  pockets  of  water!  as  he  would 
on  the  job.  The  test  taker  will  be  a little  more  alerted  than  he  will 
be  on  the  job.  because  the  test  instruction  will  tell  him  to  note  any 
observed  defects.  Nonetheless  his  response  will  be  relatively 
unalerted . 

(2)  The  job  response  is  a "critic"  response  as  is  the  test  response. 

Are  the  response  requirements  so  different  that  you  believe  A V 
simulation  will  not  yield  valid  test  results"  This  question  appears  at  this  time 
because  a "YES"  response  ends  further  consideration  of  an  A/V  simulation 
test  and  virtually  dictates  the  necessity  of  a performance  test . 

22  YES.  The  answer  selected  shows  that  the  test  responses  are  suffi- 
ciently different  from  job  responses  to  destroy  test  validity . Since  response 
options  for  written  tests  are  similar  to,  if  not  more  I'cstricted  than,  I'cspoii^n' 
options  for  A/V  testing,  a "YES"  answer  indicates  that  performance  testing  is 
necessary  for  valid  results. 

23  NO.  If  this  answer  is  selectevi  then  the  use  of  an  A/V  test  is  quite 
likely . 

EXAMPLE  - Column  4 of  Table  B-6  indicates  that  the  test  responses  to  all 
four  stimulus  variables  are  adeqviately  similar  to  provide  valid  test  results 
The  answer  to  Step  is  therefore  "NO." 

TEST  MODE  SELECTION 

GENERAL 

Up  to  this  point  it  has  been  determined  that  an  A'V  simulation  test  will 
be  considered  if  it  will  provide  a valid  test  of  at  least  one  stimulus  variable 
associated  with  a critical  task  element  At  this  time  it  is  necessary  to  decide 
whether  or  not  to  use  the  A V test  mode.  In  making  this  decision  one  should 


Specity  the  correct 

stimulus  variable  when  it  is  observed 

in  the  job  environment 


Specity  the  possible  tv^»es  ot  restnjnse 
to  each  critical  stimulus  m the  test 
setting  and  select  the  most  appropriate 


Compare  the  on  the  job  versus  the  test 
lespoi^se  requirements  to  each  stimulus 
variable  and  evaluate  the  etiects  of 
differences 


Are  the  response  requirements  so  different 
that  you  believe  that  A V simulation  will 
not  yield  valid  results^ 


Figure  B-4.  Response  Realism 
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consider  factors  such  as  cost,  motivation,  the  feasibility  of  testing  selected 
stimulus  variables  and  not  testing  others,  or  of  testing  different  stimulus 
variables  by  different  test  modes.  Figure  B-5  (Steps  24  through  27)  pro- 
vides the  sequence  of  actions  followed  to  select  the  proper  test  mode. 


SIMULATION  ALGORITHM 


i 


■I 


Consider  the  advantages  and  disadvantages  of  each  test  mode  and 
assign  a test  mode  or  test  modes.  In  assessing  cost,  it  is  necessary  to  know 
specific  characteristics  of  the  test.  In  general,  performance  tests  are  most 
expensive.  A/V  or  written  tests  supported  by  extensive  drawing  or  photog- 
raphy will  be  similar  in  cost,  and  straight  written  tests  are  least  expensive. 
Regarding  test  taker  motivation,  undesirable  effects  occur  sometimes  when 
test  stimuli  and  response  requirements  lack  realism.  In  simple  terms,  if  a test 
does  not  look  valid,  a test  taker  will  often  feel  it  is  not  fair.  This  then 
lowers  motivation. 

Motivation  may  also  be  negatively  affected  for  some  test  takers  by  the 
amount  of  reading  that  is  required . Many  Americans  today  have  reading 
problems,  and  tests  which  require  reading  skills  are  difficult  for  them  without 
regard  to  subject  content.  The  problem  reader  may  react  negatively  to  the 
written  test. 

Frequently  A/V  simulation  will  be  appropriate  for  testing  some  stimulus 
variables  and  critical  elements  but  not  for  testing  all  components  of  the  task. 
In  such  cases  the  following  options  exist: 

(1)  Select  the  single  test  mode  that  is  appropriate  to  the  most  highly 
critical  elements  and  stimulus  variables  and  test  all  critical  elements 
via  that  mode.  The  option  still  requires  that  the  mode  which  is 
used  provide  a valid  test  for  all  components  which  are  included. 

(2)  Test  critical  elements  or  stimulus  variables  via  different  testing 
modes.  Usually,  if  performance  tests  are  used,  neither  written  nor 
A/V  simulation  will  be  needed;  but  on  occasion  it  may  be  desirable 
to  probe  more  deeply  into  decision  or  perceptual  components  of  a 
task  by  supplementing  the  performance  test  with  a written  or  A/V 
simulation  test. 

(3)  Use  the  A/V  simulation  to  test  isolated  stimulus  variables.  While 
some  job  context  may  be  included  in  the  simulation , responses  will 
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be  made  to  only  the  designated  stimulus  variables.  Performance  on 
this  type  of  test  cannot  be  generalized  to  performance  of  the  task. 

The  end  product  of  this  step  is  a recommendation  of  testing  mode(s)  for 
each  stimulus  variable  and  for  the  critical  element  as  a whole. 


EXAMPLE  - Column  5 of  Table  B-6  presents  recommendations  of  TV  tests  for 
four  of  five  stimulus  variables  and  for  the  entire  critical  element.  Note  that  in 
this  case  it  has  been  decided  that  an  adequate  test  of  a critical  element  is 
possible  even  if  every  stimulus  variable  isn't  tested;  also  that  it  is  not 
recommended  that  an  attempt  be  made  to  test,  "uniformity  of  color  of  concrete" 
via  TV.  This  is  because  it  is  strongly  suspected  that  a TV  presentation  of  this 
stimulus  variable  will  be  quite  degraded  and  have  an  adverse  effect  on  test 
taker  motivation.  Thus  no  test  of  that  stimulus  element  is  preferred  to  an 
inappropriate  test.  In  considering  this  particular  critical  element,  it  is  likely 
that:  (1)  it  will  be  more  expensive  to  use  a performance  test  for  this  critical 
element  alone  since  many  concrete  slabs  will  have  to  be  poured  to  fulfill  the  test 
requirement  and  (2)  a written  test  though  less  expensive  would  be  much  less 
realistic  and  probably  less  valid. 


Has  A/V  simulation  been  selected?  This  question  is  asked  at  this 
point  because  a "NO"  answer  eliminates  the  need  for  further  concern  with  A/V 
simulation.  A "YES"  answer  leads  to  further  consideration  in  the  A/V  simulation 
test  development.  The  answer  in  this  step  is  dictated  by  the  results  of 


Step 

@ 

26 

YES.  If  this  answer  is  selected,  the  decision  is  to  develop  an  A/V 
simulation  test  and  leads  to  other  considerations  in  test  development.  It  is  now 
necessary  to  proceed  to  Step 
NO. 


27 


If  this  answer  is  selected,  an  A/V  test  is  determined  inappro- 
priate and  it  is  necessary  to  develop  written  and/or  performance  tests  for  the 
particular  element. 


REPRESENTATION  OF  JOB  CONTEXT 


GENERAL 


Earlier  evaluations  have  determined  that  A/V  simulation  will  probably 
provide  an  adequate  scoreable  unit.  However,  the  production  of  an  A/V  test 


Figure  B-5.  Test  Mode  Selection 


is  costly  and  should  not  be  used  unless  it  will  provide  a reliable  and  valid 
test.  To  be  certain  it  will  accomplish  what  is  desired,  the  task  must  be 
viewed  in  the  job  environment,  the  events  studied  and  a determination  made 
of  how  or  if  such  events  can  adequately  be  presented. 

Figure  B-6  (Steps  28,  29,  and  30)  depicts  a sequence  of  operations  that 
will  determine  the  adequacy  of  A/V  simulation  for  the  selected  task. 

SIMULATION  ALGORITHM 

(2^  Study  the  job-functional-context  in  which  this  task  is  per- 
formed and  identify  events  and  stimuli  that  are  present  and  that  may  be 
related  to  task  performance.  Up  to  this  point,  attention  has  been  narrowly 
focused  upon  stimuli  and  responses  that  are  immediately  identified  by  ana- 
lyzing the  statement  of  the  task  and  its  critical  elements.  Now  it  is  necessary 
to  study  the  context  in  which  the  task  occurs  and  identify  other  events  and 
stimuli  that  influence  performance  of  the  task. 
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Some  of  the  events  that  may  occur  which  would  influence  performance  in 
placing  and  finishing  concrete  are,  for  example;  wind  conditions,  rain,  or  hot 
or  cold  temperatures.  Therefore,  such  events  must  be  considered  as  part  of 
the  job  environment.  To  recognize  the  events,  certain  stimuli  must  be  present 
(for  example;  temperature  is  recognized  by  either  a scale  reading  on  the 
thermometer  or  warmth  and  cold  as  felt  by  the  body;  wind  conditions  may 
be  recognized  by  observing  objects  blowing). 

In  accomplishing  this  step  at  least  two  subject  matter  experts  should 
observe  a qualified  person  perfom  the  task  in  a typical  job  setting.  During 
this  observation  attention  should  be  focused  upon  the  effect  that  people  and 
conditions  (events)  have  upon  task  performance.  The  end  product  should  be 
a list  of  contextual  events  and  stimuli  that  should  te  considered. 


EXAMPLE  - The  critical  element,  "Direct/Control  Finishing  Concrete  Using  a 
Wood  Float",  is  influenced  by  a number  of  variables  that  have  not  been  con- 
sidered previously.  These  are  the  events  and  stimuli  that  influence  the  job 
performance.  A partial  list  has  been  entered  in  Columns  1 and  2 of  Table  B-7. 
This  list  provides  the  basis  for  the  following  steps . 


Determine  the  importance  of  each  contextual  event  to  task  perfor- 
mance. The  purpose  of  this  step  is  to  further  define  the  simulation 
requirements.  Experience  has  shown  that  it  is  not  necessary  to  simulate  total 
environment  for  valid  use  of  simulation  in  training  or  testing,  but  it  is 
necessary  to  include  stimuli  that  relate  directly  to  the  individual's  perfor- 


mance. The  same  basic  question  presented  earlier  in  Step  is  repeated 

in  this  step.  In  this  step  it  is  necessary  to  review  the  list  of  contextual 
events  which  were  developed  in  Step  and  to  determine  their  impor- 

tance to  task  performance.  "Considering  all  of  the  information  which  is 
provided  by  all  of  the  contextual  events  which  are  typically  present  when  this 
critical  element  is  performed,  how  often  does  this  one  contextual  event  pro- 
vide unique  information  that  is  essential  to  proper  perfonnance  of  the  task 
element?" 

An  event  will  be  very  important  if  it  always  provides  unique  information 
which  is  essential  to  performance.  It  will  be  less  important  if  it  either  dupli- 
cates information  already  available  or  if  its  informational  value  has  a less 
direct  effect  upon  proper  performance.  Each  contextual  event  should  be 
labeled  as  ver^  important . moderately  impoj'tajnt,  or  not  very  important. 
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TABLE  B-7.  EVALUATION  OF  THE  JOB  FUNCTIONAL-CONTEXT 
Critical  Element:  Direct/Control  Finishing  Concrete  Using  a Wood  Float 


(1) 

Contextual 

Event 

(Step  @ ) 

(2) 

Contextual 
Stimuli 
(Step  @ ) 

(3) 

Criticality  of 
Contextual  Event 
(Step  (g)  ) 

(4) 

Inclusion  of  Event 
in  Test 

(Step  (g)  ) 

(5) 

Stimuli  Selected 
for  Presentation 
(Step  ) 

Type  of  concrete 

Oral  and  written 
statement 

Very  important 

Yes 

Oral  and  written 
statement 

Temperature 

Thermometer, 
warmth  or  cold 
as  felt 

Very  important 

Yes 

Thermometer  plus 
oral  statement 

Humidity 

Rain,  humidity 
as  felt 

Very  important 

Yes 

Written  plus  oral 
statement 

Wind  speed 

Wind  as  felt  by 
body,  observa 
tion  of  items 
blowing 

Very  important 

Yes 

Observation  of 
flag  blowing 
plus  oral  statement 

Time  since 

placing 

concrete 

Watch  face, 
performance  of 
other  tasks  of 
known  duration 

Very  important 

Yes 

Watch  race,  oral 
statement 

EXAMPLE  - In 

performing  this  step  the  five  contextual  events  which  are 

listed  in  Column  1 of  Table  B-7  are  reviewed.  A judgment  is  made  of  the 
importance  of  each  event  in  perfonming  the  critical  element . In  Column  3 of 
Table  B-7,  this  judgment  is  recorded.  A contextual  event  is  rated:  (1) 
very  important  if  it  provides  unique  information  that  is  essential  to  proper 
performance,  (2)  moderately  important  if  it  provides  inforroation  which  is 
either  unique  or  essential,  but  not  both,  and  (3)  not  very  impo^Tant  if  it  pro- 
vides information  which  is  neither  unique  nor  essential  to  proper  performance. 

Identify  which  contextual  events  shall  be  included  and  how  they  will 
be  presented.  All  very  important  events  should  be  represented  in  the  test. 
Moderately  important  or  not  very  important  events  may  be  included  to  the 
degree  that  the  added  realism  is  compatible  with  cost  and  time  constraints. 
Often  contextual  information  cannot  be  pr’esented  in  real  time  or  with  a high 
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degree  of  stimulus  realism,  but  providing  the  information  in  oral  or  written 
form  may  still  enhance  the  validity  of  the  test.  It  is  also  noted  that  many 
events  are  experienced  in  terms  ,of  a number  of  related  stimuli . 


EXAMPLE  - In  this  example  all  of  the  contextual  events  which  are  identified 
in  Step  are  judged  very  important  in  Step  Thus  all  are 

included.  Since  most  events  in  real  life  are  represented  by  a number  of 
stimuli,  it  is  desirable  to  provide  redundant  stimuli  on  the  test.  Column  5 of 
Table  B-7  requires  oral  plus  visual  stimuli  in  presenting  each  contextual 
event  to  increase  the  probability  that  these  events  will  be  recognized. 


FINAL  ASSESSMENT  OF  PRESENTATION  REALISM 


GENERAL 

All  simulation  requirements  have  now  been  specified.  Before  developing 
the  test,  it  is  advisable  to  check  any  doubts  about  the  realism  with  which  any 
of  the  stimulus  variables  can  be  presented  via  the  simulation . This  section 
specifies  the  steps  to  be  taken  if  there  are  any  concerns.  The  sequence  of 
actions  is  shown  in  Figure  B-7. 

SIMULATION  MODEL  ALGORITHM 


Are  you  certain  that  the  A/V  simulation  will  provide  an  adequate 
presentation  of  all  important  stimulus  variables?  To  get  to  this  step,  the 
adequacy  of  A/V  simulation  has  been  judged  at  least  probable.  This  question 
is  inserted  to  encourage  the  production  and  evaluation  of  the  stimulus  vari- 
ables whenever  there  is  doubt  about  the  adequacy  of  the  A/V  presentation . 
A high  degree  of  certainty  may  exist  when  one  has  previously  produced 
or  observed  A/V  simulation  of  some  stimulus  variable,  but  when  experience  is 
absent  a prototype  tape  will  be  worthwhile. 
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Study  tht  job-functtonal  conttxt  in 
wthich  this  task  it  parformtd  and 
idantity  avtnts  and  stimuli  that  art 
prMtnt  and  that  nWV  ba  ralatad 
to  task  partormanca. 


Datarmina  tha  importanca  o1  each 
contextual  event  to  task  performance. 


Identify  which  contextual  events  shill  be 
included  in  the  test  and  how  they  will  be 
presented. 


Figure  B-6.  Represerrtation  of  Job  Context 

32  NO.  If  this  answer  is  selected  in  response  to  the  question  asked  in 

Step  <3^  , it  will  require  the  development  of  a prototype  tape  that  will 

enable  a judgment  of  the  adequacy  of  simulation  of  specific  stimulus  variables . 

EXAMPLE  - One  may  not  be  certain  that  TV  adequately  represents  the  effect  of 
placing  a slight  pressure  on  concrete  before  and  when  it  is  ready  for  floating. 
If  this  is  the  case,  development  of  the  test  is  premature  and  a prototype  tape  is 
recommended . 

33  If  the  answer  to  this  question  is  "YES" , proceed  to  Step 

Are  the  stimulus  variables  presented  with  adequate  reality?  At  least 
two  subject  matter  experts  should  be  involved  in  answering  this  question . The 
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answer  will  be  subjective  The  main  consideration  in  answering  is  whether  the 
quality  of  simulation  is  such  that  a test  taker  will  either  (1)  miss  an  item 
because  the  stimulus  variable  is  too  ambiguous  or  unrealistic  or  (2)  get  an  item 
right  because  the  simulated  stimulus  is  too  obvious.  If  it  is  suspected  that  the 
quality  of  simulation  will  either  increase  or  decrease  the  probability  of  a correct 
response  to  the  stimulus  variable,  simulation  is  not  adequate. 


EXAMPLE  - If  a prototype  tape  shows  one  applying  a slight  pressure  to  the 
concrete,  the  subject  matter  experts  must  decide  whether  the  portrayed 
response  of  the  concrete  to  the  pressure  is  as  clear  as  when  looking  at  the 
actual  concrete.  He  must  ask  himself  the  following:  If  it  is  not  clear,  is  it  bad 
enough  to  confuse  the  lest  taker?  If  it  is  clearer  than  what  is  typicaUy 
observed,  will  this  result  in  more  test  takers  doing  better? 
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or  this 


YES.  If  a "YES"  answer  is  selected  in  either  Step 
step,  it  wUl  establish  the  appropriateness  of  the  intended  mode  of  A/V 
simulation  and  it  is  time  to  begin  developing  the  test 

If  this  answer  is  chosen,  it  rules  out  the  intended  mode  of 
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A/V  simulation  and  directs  the  use  of  perfomance  or  written  tests. 


TEST  DEVELOPMENT 


GENERAL 

The  preceding  steps  have  provided  a detailed  procedure  for  selecting  and 
detennining  the  adequacy  of  A/V  simulation  as  a mode  of  testing  a specific  task 
element.  Figure  B-8  (Steps  37  through  54)  presents  the  sequence  of  operations 
required  to  develop  the  test.  This  consists  of  selecting  the  sequence  of  critical 
elements,  writing  the  audio-visual  script,  developing  scoring  procedures  and 
determining  the  reliability  and  validity  of  the  test 


SIMULATION  ALGORITHM 


Sequence  the  selected  critical  elements  in  tenns 
An  A/V  simulation  test  may  include  a number  of  items  or 


of  test  items, 
task  elements. 
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Figure  B 7.  Final  Assessment  of  Presentation  Realism 

Ordinarily,  on  the  job,  critical  elements  occur  in  a definite  order  within  task 
elements,  and  this  order  should  be  preserved.  Sometimes  the  simulation  test 
may  include  critical  element  items  fi-om  two  or  more  tasks.  In  this  case  both 
tasks  and  the  critical  elements  should  be  sequenced  as  they  occur  on  the  job. 

(3^  Write  audio-visual  script  for  the  simulated  A/V  test  presentation. 
This  is  the  first  step  in  test  construction.  In  Step  (3^  the  critical 
elements  to  be  tested  were  sequenced  - this  identified  what  has  to  be  tested 
and  when  Now  it  is  necessary  to  specify  the  exact  audio  and  visual  content 
of  the  test  How  much  and  what  parts  of  the  critical  element  will  be  pre- 
sented in  each  test  item  and  what  will  the  test  taker  be  tolil  via  audio  about 
the  critical  element  and  his  response?  This  task  requires  a media  specialist , a 
subject  matter  expert  and  somt'one  who  can  write  the  audio  script  in  simple 
and  unambiguous  technically  accurate  lan^uafje.  Three  individuals  will  not  be 
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requirtHl  if  one  person  has  skills  in  at  least  two  of  the  areas.  Having  these 
resourees  present  in  the  initial  eonstruetion  of  the  test  item  should  reduoe 
the  frequeney  of  the  following  types  of  pix^blems 


(.1)  The  subject  matter  expert  states  requirements  for  a presentation 
which  cannot  be  presented  via  the  media. 

(2)  The  media  expert  fails  to  optimize  his  use  of  the  media  because  he 
does  not  know  the  relative  importance  of  various  parts  of  the 
stimulus  field. 

i,3)  The  audio  script  is  not  optimally  coordinated  with  video. 

(4>  The  audio  script  detracts  fixim  test  validity  because  it  is  either 
technically  accurate  but  too  complex  for  the  test  taker  or  very  clear 
and  simple  but  not  technically  accurate. 

A few  general  principles  that  should  be  remembered  in  perfonning  this 
step  are: 

(1)  Present  visual  test  stimuli  fi-om  the  reference  of  the  person  perfomi- 
ing  the  task.  A picture  of  someone  disassembling  a rifle  presents  a 
different  set  of  stimuli  than  does  a picture  of  the  rifle  being  disas- 
sembled fi'om  the  view  of  the  person  who  disassembles  the  rifle.  A 
test  item  on  assembly  and  disassembly  of  a rifle  will  ideally  present 
the  stimuli  as  they  are  perceived  by  the  person  doing  the  task 
rather  than  as  they  are  viewed  by  an  observer.  When  a critical 
element  pertains  to  supervising  others,  the  observer's  reference  will 
be  appropriate. 

(2>  The  effect  of  stimuli  not  specifically  relevant  to  the  test  item  must 
be  carefully  evaluated.  The  elimination  of  all  apparently  unrelated 
stimuli  may  destroy  the  job  context  and  result  in  a videt>  presenta- 
tion that  lacks  realism.  Also,  sometimes  a stimulus  that  should  be 
trivial  and  insignificant  can  be  highlighted  in  the  simulation  and 
become  very  distracting.  For  example,  test  takers  may  pay  more 
attention  to  a soldier's  cap  being  on  cxwked  than  to  the  manner  in 
which  he  is  performing  a task.  The  script  for  the  vidtv  should 
thus  describe  the  video  requirements  in  considerable  detail,  to 
include  stimuli  which  provide  context  and  exclude  stimuli  that  may 
distract 
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t: 


(4) 


(5) 


(3)  In  the  job  enviix)ninent  the  soldier  has  a jfreater  opportunity  to 
obtain  and  verify  infomation  than  he  does  while  taking  an  A V 
simulation  test.  For  example,  he  can  take  a second  Uxik  or  ask  a 
buddy  if  he  sees  something  in  the  same  way . The  test  taker  will 
not  have  this  freedom  to  verify  infomiation  on  his  own  initiative  in 
the  A/V  simulation  test , This  emphasizes  the  requirement  for  suf- 
ficiently lengthy  and  clear  presentations  of  all  important  stimulus 
variables  and  suggests  that  repeated  exposure  of  some  stimulus 
variables  may  add  to  test  validity. 

The  script  must  integrate  response  requirements  into  the  A/ A’  tests 
The  video  need  not  shut  off  when  the  test  taker  is  supposed  to 
respond,  but  the  script  should  insure  that  the  test  taker  knows 

when  and  how  to  respond  and  that  he  will  not  miss  other  infor- 

mation while  responding 

Scoring  procedures  require  at  least  two  responses  to  each  critical 
element.  In  general  more  than  two  responses  are  preferred. 

Develop  criterion  referenced  scoring  pivcedures  in  accordance 
with  the  Manual  for  Developing  SQTs.  In  perfoniiing  this  step,  it  is  neces- 
sary to  refer  to  the  current  guidance  for  developing  SQ'l's . There  is  one 

important  variation  to  note.  The  guidance  uses  the  task  as  the  basic  behav- 
ioral unit  for  scoring.  In  scoring  performance  on  the  A/V  simulation  test, 
the  critical  element  is  the  basic  behavioral  unit.  In  using  the  procedures 
that  are  clearly  spelled  out  in  the  manual,  simply  apply  the  ptwedures  to  the 
critical  element  instead  of  the  task.  For  example,  each  critical  element  must 
have  two  or  more  test  items  associated  with  it  and  each  critical  element  will  be 
scored  on  a go/no-go  basis. 

(4^  Review  test  script,  format  and  scoring  with  five  other  subject 
matter  experts.  Before  developing  the  test,  it  is  desirable  to  obtain  evalu- 
ative comments  ftx'm  qualified  people  who  were  not  involved  in  constructing 
the  test . They  should  review  the  audio  and  vidw  script  format  and  scoring 
procedures  and  judge  whether  the  test  is  technically  sound  and  appears 
capable  of  providing  a valid  measure  of  ability  to  perfoim  the  critical  ele- 
ments. In  orienting  the  reviewers,  it  is  necessary  to:  (.1)  clearly  identify 
the  purpose  of  the  test,  e g.,  which  critical  elements  are  being  measured, 
and  (2)  to  emphasize  that  reviewers  should  focus  on  technical  accuracy  and 
clarity  of  presentation.  Five  subject  matter  experts  should  individually 
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review  the  proposed  test  materials,  without  further  explanation  or  interpreta- 
tion from  the  script  writers  They  should  record  all  apparent  inaccuracies 
or  ambiguities,  Following  this,  the  writers  should  discuss  each  reviewer's 
comments  with  hmi  The  script  writers  should  attempt  to  modify  the  script 
format  or  scoring  procedures  to  the  satisfaction  of  each  reviewer  Remember, 
if  the  subject  matter  expert  made  an  "inappropriate"  comment  because  he 
didn't  understand  the  test  item,  the  more  naive  test  taker  may  well  miss  the 
item  because  it  is  also  ambiguous  for  him 

It  is  often  more  difficult  for  script  writers  to  participate  in  this  review 
than  it  is  to  write  the  script  for  them.  Script  writers  should  keep  in  mind 
that . 


(2) 


(3) 


U)  If  script  requires  your  explanation  now,  it  will  probably  also 
require  explanation  to  some  test  taker.  But  - you  will  not  be  able 
to  do  that.  It's  better  to  be  safe  and  modify  the  script  now. 

The  subject  matter  expert  is  not  attacking  you  or  your  technical 
knowledge  or  your  writing  ability  if  he  suggests  a change.  If  a test 
comes  out  of  this  review  step  without  any  modifications,  it  is  more 
likely  a sign  of  sloppy  review  than  of  a perfectly  constructed  test 
The  time  taken  to  revise  at  this  point  in  the  test  development  is 
minor  compared  to  that  required  to  modify  the  A simulation  test 
because  of  inadequate  validity  or  reliability.  If  in  doubt,  make 
changes  at  this  stage  of  review . 

(.4)  This  step  should  end  with  all  subject  matter  experts  agreeing  that 
when  developed,  the  test  should  provide  a technically  accurate  and 
valid  measure  of  the  critical  elements . 

Is  the  test  judged  technically  sound'.’  This  question  is  asked 
explicitly  because  of  its  importance  and  because  an  objective  basis  for 
answering  is  provided.  In  practice  this  step  occurs  simultaneously  with  the 
conclusion  of  Step  Test  developers  will  perform  this  step  as  they 

review  comments  from  and  discussions  with  each  subject  matter  expert 

YES.  If  this  answer  is  selected  following  the  preceding  review 
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and  revision  and  the  subject  matter  experts  agree  on  the  accuracy  and  clarity 
of  the  test  items,  it  is  anticipated  that  a test  will  be  acceptable  in  terms  of 
validity  and  reliability . 

43  NO.  This  answer  may  be  selected  if  one  subject  matter  expert 
contends  that  an  item  is  technically  inaccurate  or  unclear  to  him  A "NO" 


answer  is  not  required  if  the  subject  matter  expert  agrees  to  item  accuracy 
and  clarity  but  feels  that  the  item  should  be  presented  or  scored  in  another 
manner.  An  answer  of  "NO"  requires  a return  to  Step 

(S)  Develop  A/V  simulation  test  and  response  forms.  The  A/V  simu- 

lation test  is  physically  developed  in  this  step.  Audio  and  video  recording 
and  editing  m'lst  be  accomplished  by  personnel  who  are  skilled  in  production 
techniques  and  procedures.  A subject  matter  expert  must  be  present  to 
assure  adherence  to  the  audio  and  visual  script  and  that  any  last  minute 
modification  will  be  technically  acceptable.  Technical  flaws  such  as  extra- 
neous recording  sounds  or  lighting  changes,  which  are  apparently  not  critical 
to  task  performance  will  often  distract  the  test  taker.  Professional  quality 
work  on  this  step  is  highly  important.  It  is  beyond  the  scope  of  this  model 
to  go  into  the  details  of  developing  the  A/V  simulation. 

(4^  Review  tests  with  five  subject  matter  experts.  In  this  review  the 
subject  matter  expert  will  first  be  given  the  test,  just  as  it  is  to  be  given  to 
other  soldiers.  Extra  instructions  or  background  will  not  be  provided.  The 
tests  will  then  be  scored  according  to  the  prescribed  scoring  procedure. 

Following  this,  the  subject  matter  experts  should  be  encouraged  to 
discuss  their  feelings  about  the  quality  of  the  test. 

Carefully  study  all  errors  on  the  test  made  by  subject  matter  experts. 
There  is  a good  chance  that  such  errors  indicate  a flaw  in  the  item . Check 
with  the  person  who  made  the  error  and  determine  the  cause. 

Record  all  subjective  comments.  Favorable  comments  are  nice  to  hear, 
but  pay  more  attention  to  the  unfavorable.  Remember,  the  ultimate  goal  is  a 
well- developed  test. 

Is  the  A/V  presentation  of  acceptable  quality?  This  step  occurs 
simultaneously  with  the  conclusion  of  Step  Test  developers  con- 

sider this  question  as  they  analyze  the  test  results  and  comments  obtained 
from  subject  matter  experts. 

[47]  YES.  This  answer  may  be  given  when..  (1)  no  errors  were  made 
by  subject  matter  experts  because  of  inaccuracies  or  ambiguities  in  the  test 
and  (2)  no  more  than  two  of  the  five  subject  matter  experts  agree  that  any 
specific  aspect  of  the  test  is  misleading,  ambiguous,  or  distracting. 

NO  This  answer  will  be  selected  whenever  it  is  found  that  (Da 
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subject  matter  expert  has  missed  a test  item  because  of  an  ambiguity  or  error 
in  the  test,  or  (2)  at  least  three  of  the  five  subject  mattt'r  experts  agree  that 
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some  aspect  of  the  A/V  simulation  is  misleading,  ambiguous,  or  distracting. 


The  rather  conservative  standard  of  three  out  of  five  is  used  because  it  is 


expensive  to  revise  the  test  at  this  stage  of  development.  An  answer  of 
"NO"  requires  a return  to  Step 

Is  the  test  of  adequate  reliability?  In  this  step  a determination  is 
made  of  whether  or  not  the  test  is  reliable.  The  principle  is  that  people  who 
fail  opce  should  fail  the  second  time  and  people  who  pass  once  should  pass 
the  second  time.  The  method  of  computing  reliability  is  contained  in  Chapter 
7 of  Developing  Criterion  Referenced  Tests  ^ Appendix  B . 

50  YES.  This  answer  is  selected  if  the  0 coefficient  is  .50  or  more. 
The  test  item  has  sufficient  reliability  for  validation. 

51  NO.  This  answer  is  selected  if  the  0 coefficient  is  .49  or  less. 
It  is  npw  necessary  to  return  to  Step 

Is  the  test  of  adequate  validity?  Earlier  steps  established  the 
content  validity  of  the  A/V  test.  Subject  matter  experts  who  possess  the 
knowledge  and  skill  of  the  MOS  have  judged  the  content  of  the  test  to  be 
adequate.  In  this  step  it  is  necessary  to  compare  individual  results  on  the 
A/V  test  with  their  results  on  a performance  test  covering  the  same  test 
items.  This  is  accomplished  in  a manner  similar  to  the  validation  procedures, 
currently  in  effect,  for  Phase  1 of  a written  test. 

The  validation  is  accomplished  as  follows : 

(1)  Validate  the  A/V  test  against  a performance  test  for  the  same  task. 
Two  or  more  experts  develop  procedures  for  administering  and 
scoring  a performance  test  of  the  task.  The  procedures  are  refined 
until  the  experts  agree  perfectly  in  scoring.  They  may  administer 
the  test  to  each  other  or  another  individual.  The  scoring  of  the 
experts  is  the  standard  in  subsequent  steps  of  this  type  of 


validation . 


(2)  Administer  both  the  performance  test  and  the  A/V  test  based  on  the 
task  to  groups  of  at  least  five  masters  and  five  nonmasters.  The 
minimum  acceptable  standard  for  the  performance  test  of  the  task  is 
that  80  percent  or  more  of  the  master  group  pass  the  performance 
test  and  that  20  percent  or  less  of  the  nonmaster  group  pass  the 
performance  test.  No  evaluators  other  than  the  expert  need 
observe  administration  of  the  performance  test. 


(3)  Obtain  the  ei^tent  of  agreement  between  go/no-go  on  the  performance 
test  and  pass-fail  on  the  A/V  test.  Sixty  percent  or  more  of  the 
scores  must  be  in  agreement;  that  is,  at  least  60  percent  of  the 
soldiers  pass  both  the  performance  test  and  the  A/V  test  or  fail  both 
the  performance  test  and  the  A/V  test.  A minimum  of  60  percent 
agreement  must  be  obtained  for  each  A/V  test  item. 

Assume  that  four  A/V  test  items  are  tried  out.  For  each  of  these  four 
items,  prepare  a table  as  shown  below.  This  is  called  a two  by  two  table,  where 
the  extent  of  agreement  is  calculated  between  the  performance  test  of  the  task 
and  each  A/V  test  item.  In  general  a table  is  interpreted  in  the  following 
manner : 


PERFORMANCE  TEST 

A/V  TEST 


ITEM 

PASS 

FAIL 

CELL  1 

CELL  2 

Pass  performance  test 

Fail  performance  test 

PASS 

and 

and 

pass  A/V  item 

pass  A/V  item 

CELL  3 

CELL  4 

Pass  performance  test 

Fail  performance  test 

FAIL 

and 

and 

fail  A/V  item 

fail  A/V  item 

Extent  of  agreement  = Cell  1 + Cell  4 

Y inn 

Cell  1 + Cell  2 + 

Cell  3 + Cell  4 
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Now  to  the  specific  examples: 


A/V 

ITEM  1 

PERFORMANCE 

PASS 

TEST 

FAIL 

Extent  of  agreement  = 

Pass 

5 

0 

t(5  + 6)  : 10)  X 100  = 100% 

Fail 

0 

5 

A/V  item  is  satisfactory. 

A/V 

ITEM  2 

PERFORMANCE 

PASS 

TEST 

FAIL 

Extent  of  agreement  = 

Pass 

4 

2 

1(4+3)  10  1 X 100  = 70% 

Fail 

1 

3 

A/V  test  is  satisfactory 

A/V 

ITEM  3 

PERFORMANCE 

PASS 

TEST 

FAIL 

Extent  of  agreement  = 

Pass 

3 

2 

1(3  + 3)  : 10 1 X 100  = 60% 

Fail 

2 

3 

A/V  item  is  satisfactory 

A/V 

ITEM  A 

PERFORMANCE 

PASS 

TEST 

FAIL 

Extent  of  agreement  = 

Pass 

3 

3 

[ (3  + 2)  + 10  1 X 100  = .S0% 

Fail 

1 

2 

A/V  item  is  unsatisfactory 

Because  Items  1 , 2 and  3 had  sufficient  agreement  with  the  performance  test  of 
the  task,  60  percent  or  more,  they  were  satisfactory.  Since  Item  4 did  not  meet 
the  60  percent  criterion,  it  is  unsatisfactory,  and  therefore,  either  requires 
revision  or  replacement  by  a satisfactory  item.  Complete  this  procedure  of 
comparing  A/V  items  based  on  a task  to  a performance  test  of  that  task  for  each 
of  the  A/V  items. 

YES.  If  the  test  has  adequate  validity,  the  test  development  is 

complete . 

[m]  no.  This  answer  will  be  selected  if  the  test  is  not  detennined  valid 
ind  the  test  developer  must  go  back  to  Step  (3^ 
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Sequence  the  selected  critical  elements 
in  terms  of  test  items 


Write  audio  and  visual  script  for  the 
simulated  A/V  test  presentation 


Develop  criterion-referenced  scoring 
procedures  in  accordance  wah  the 
Manual  for  Developing  SQTs 


Review  test  script,  format.  arxJ 
scoring  with  five  other  subiect 
matter  experts 


Is  the  test  judged  technically  sound? 


Appendix  C 

SIMULATION  TEST  AUDIO  SCRIPT  AND 
ANSWER  SHEET 


APPENDIX  C 


SIMULATION  TEST  AUDIO  SCRIPT  AND  ANSWER  SHEET 

AUDIO  SCRIPT 

This  is  an  P’xpt'rimenlal  Skill  Qualification  Test,  S-Q-T  two,  for  M-O-S  51B. 

You  have  completed  your  training  and  now  you  are  ready  to  be  tested  on  the 
tasks  "Constructing  and  Erecting  Wall  Forms"  and  "Placing  and  Finishing 
Concrete . " 

This  test  has  been  divided  into  three  units. 

Unit  1:  Handtool  Maintenance  and  Materials  Preparation. 

Unit  II;  F.recting  Wall  Forms. 

Unit  111;  Placing  and  Finishing  Concrete. 

The  format  for  answering  the  test  items  will  change  from  one  unit  to  another, 
and  you  will  be  given  directions  telling  you  how  to  mark  your  answer. 

However,  for  every  item  on  the  test  you  will  be  shown  a situation  and  a 
question  will  be  stated.  Then  you  will  be  given  several  answers  from  which 
to  choose,  and  the  question  will  be  repeated.  You  will  be  given  time  to 

circle  your  answer. 

You  should  have  an  answer  sheet  on  your  desk.  If  you  do  not,  raise  your 
hand  and  one  will  be  given  to  you . 

i 

Look  at  the  top  right-hand  corner  of  your  answer  sheet.  You  should  have 

filled  in  your  name,  social  security  number  and  paygrade.  Be  sure  you  have  | 

completed  this  information  before  turning  in  your  answer  sheet.  j 

I 

1 

Now  look  at  the  section  labeled  "Unit  1"  on  your  answer  sheet.  j 

I 


1 
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Also  there  are 


Notice  that  in  each  item,  there  are  up  to  four  letter  choices, 
numbers  one  through  five  to  indicate  "Safety  Violations." 


book  in  the  upper  left-hand  corner  of  your  answer  sheet.  There  are  Hve 
possible  Safety  Violations; 

1 . Failure  to  ground  electric  tools  or  equipment  properly . 

2.  Failure  to  wear  protective  gear  when  necessary. 

3.  Use  of  a tool  in  a hazardous  manner. 

4.  Unsafe  vehicle  operating  procedures, 
and 

5.  Unsafe  material  handling  or  storage  procedures. 


If  at  any  time  during  the  test  you  see  any  of  these  five  Safety  Violations, 
circle  the  number  that  corresponds  with  the  Safety  Violation  beside  that  item. 


You  will  n^  be  told  when  to  look  for  Safety  Violations!  So  be  alert.  Watch 
for  them  in  Units  one  and  two. 


Now  let's  look  at  two  sample  test  items. 

Sample  number  one.  You  are  checking  plywood  to  see  if  it  is  usable  for 
building  a wall  form.  Another  member  of  the  team  is  cutting  studs  to 
length . 

To  which  side  of  the  plywood  do  you  nail  the  studs? 

A. 

B. 


To  which  side  of  the  plywood  do  you  nail  the  studs?  Now  mark  your  answer. 

You  should  have  circled  letter  "A",  for  sample  item  one  on  your  answer 
sheet.  Side  "A"  is  the  rough  side,  therefore,  it  is  the  side  to  which  you 
would  nail  the  studs.  Did  you  notice  the  Safety  Violations?  The  man  using 
the  saw  was  not  wearing  goggles. 
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You  should  have  ciroled  number  two  indicating  failure  to  wear  protective 
gear  when  necessary . 

Now  let's  look  at  sample  item  number  two.  You  are  checking  tools  to  see  if 
they  are  usable. 

Which  ttx>l  is  not  U’«able? 

A. 

B. 

C. 

D. 

Which  tool  is  ncd  usable?  Now  mark  your  answer. 

When  you  noticed  that  the  handsaw  was  bent,  you  should  have  circled  "C" 
for  sample  item  number  two  on  your  answer  sheet. 

Since  there  were  no  Safety  Violations  you  should  not  have  circled  any  of  the 
numbers  for  that  time. 

Many  steps  are  necessary  to  prepare  for  building  I'oncrete  wall  forms.  Nine 
of  these  steps  will  be  included  in  this  first  unit.  Let's  begin  the  test. 

Item  one.  Jointing  a saw  ensures  a clean  cut.  To  joint  the  saw,  it  must  be 
placed  in  a vise.  For  jointing,  which  saw  is  ix>rrectly  gripped  in  the  vise? 

A. 

B. 

C. 

D. 

For  jointing,  which  saw  is  correctly  gripped  in  the  vise?  Now  mark  your 
answer. 


\2\ 


Item  two.  Which  man  is  correctly  jointing  the  saw  teeth? 


Which  man  is  correctly  jointing  the  saw  teeth?  Now  mark  your  answer 

Item  three.  Which  man  has  properly  gripped  the  hamlsaw  in  the  vise  and  is 
correctly  sharpening  it? 

A. 


Which  man  has  properly  gripped  the  handsaw  in  the  vise  and  is  correctly 
sharpening  it?  Now  mark  your  answer 

Item  four.  You  are  building  a wall  form.  Braces  for  this  wall  fom  retiuire 
fourteen  "sixty-seven  inch"  lengths  of  two-by-four  In  order  to  save  on 
materials,  which  stack  of  two-by-fours  would  you  use" 


Braces  for  this  wall  form  require  fourteen  "sixty-sevei\  iiuh"  U-ngths  v'f  two- 
by-four.  In  order  to  save  on  materials,  which  st.ick  v'f  two- by -fours  would 
you  use? 

Now  mark  your  answer. 

Item  five.  From  the  stack  you  selected,  how  many  boards  will  vou  take  to 
cut  fourteen  "sixty-seven  inch"  lengths" 


From  the  stack  you  selected,  how  many  boards  will  you  take  to  cut  fourteen 
"sixty-seven  inch"  lengths?  Now  mark  your  answer . 

Item  six.  Which  man  is  correctly  marking  "twelve-inch  centers"  on  the 
two-by-four? 

A. 

B. 

C. 

D. 

Which  man  is  correctly  marking  "twelve  inch  centers"  on  the  two-by-four? 
Now  mark  your  answer. 

Item  seven.  Look  at  the  information  given  on  this  drawing.  What  is  the 
correct  spreader  length^ 

A. 

B. 

C. 

D. 

Look  at  the  information  given  on  this  drawing.  What  is  the  correct  spreader 
length?  Now  mark  your  answer. 

Item  eight.  Look  at  these  spreaders.  Which  one  should  you  choose  for 
constructing  this  wall  form? 

A. 

B. 

C. 

D. 

Which  one  should  you  choose  for  constructing  this  wall  form?  Now  mark  your 
answer. 
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Item  nine.  Choose  the  correct  angle  between  the  saw  and  the  work. 


Choose  the  correct  angle  between  the  saw  and  the  work.  Now  mark  your 


answer. 


Item  ten.  Which  nails  are  best  suited  for  formwork? 


Which  nails  are  best  suited  for  formwork?  Now  mark  your  answer. 


Item  eleven.  Which  example  shows  the  correct  method  of  hammering"; 


Which  example  shows  the  correct  method  of  hammering?  Now  mark  your 


answer. 


This  completes  Unit  1 


Unit  11  will  test  you  on  your  ability  to  recognize  the  pivper  method  for 
erecting  the  wall  form. 


lAX>k  at  Unit  11  on  your  answe'r  sheet.  The  items  listed  there  will  be  shown 


to  you  in  a continuous  sequence. 
Wall  plumb 


Walls  level 


I 


% 

Tie  wires:  holes  drilled  correctly 

Spreader  properly  drilled  < 

Wale  properly  installed 

J. 

Spreaders  in  the  correct  position  >; 

Tie  wires:  twisted  correctly  | 

And,  keyway  positioned  correctly.  You  are  to  judge  whether  each  of  the  I 

items  on  the  list  is  performed  correctly  or  incorrectly  and  circle  the  letter 
"C"  for  correct,  or  the  letter  "1"  for  incorrect  in  the  appropriate  space. 

Remember,  look  for  Safety  Violations. 

Watch  carefully.  You  will  be  shown  this  sequence  only  once.  This  time, 
you  must  circle  your  answer  as  each  item  is  presented.  Please  note  that 
each  item  is  to  be  marked  C-correct  or  1-incorrect.  Let's  begin. 

Item  twenty.  Here  are  four  methods  for  bracing  a wall  form.  You  will 
be  shown  the  methods  twice.  The  first  time  is  for  observation  only.  After 
the  second  viewing  you  will  be  given  time  to  circle  your  answer.  Select  the 
best  method  for  bracing  a wall  form. 

A. 

B. 

C. 

D. 

Look  at  the  braces  again.  Select  the  best  method  for  bracing  a wall  form. 

A. 

B. 

C. 


j Now  mark  your  answer 

1 

This  completes  Unit  11 

Unit  111  will  test  you  on  your  ability  to  recognize  the  correct  placing  and 
finishing  of  concrete 


i:s 


lxx)k  at  Unit  111  on  your  answer  sheet.  This  Unit  will  be  completed  in  the 
same  manner  as  in  Unit  1.  Circle  the  letter  you  choose  as  the  correct 
answer  for  each  item.  There  are  no  Safety  Violations.  Let's  bejfin. 

Item  twenty-one.  Concrete  will  be  poured  into  a form  forty-eight  inches 
high.  At  what  level  should  you  begin  to  vibrate  the  concrete*^ 

A.  6 inches. 

A 

B . 18  inches . 

C.  36  inches. 

D . 48  inches . 

At  what  level  should  you  begin  to  vibrate  the  concrete?  Now  mark  your 
answer. 

Item  twenty-two.  Which  is  the  correct  way  to  store  a wheelbarrow? 

A. 

B. 

C. 

D. 

Which  is  the  correct  way  to  store  a wheelbarrow?  Now  mark  your  answer. 

Item  twenty-three.  Choose  the  proper  screeding  technique.  You  will  see  each 
choice  only  once. 

A. 

B. 

C. 

D. 

Choose  the  proper  screeding  technique.  Now  mark  your  answer. 

Item  twenty-four.  Is  this  concrete  ready  to  be  floated?  You  will  see  each 
choice  only  once. 

A.  Yes. 

B.  No. 

C.  The  information  is  insufficient. 
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Item  twenty-five.  Which  is  the  proper  method  for  floating  concrete?  You  will 
see  each  choice  only  on^. 


A. 

B. 

C. 


I i'  I 

I :■'! 

* -if} 


ifi' 

,>r 

t 


Which  is  the  proper  method  for  floating  concrete?  Now  mark  your  answer. 

Item  twenty-six.  Which  slab  has  been  properly  floated?  You  will  see  each 
choice  only  once. 

A. 

B. 

C. 

D. 

Which  slab  has  been  properly  floated?  Now  mark  your  answer. 

Item  twenty-seven.  Look  at  this  concrete.  Is  it  ready  for  first  troweling? 

A.  Yes. 

B.  No. 

C.  The  information  is  insufficient. 

Is  It  ready  for  first  troweUng?  Now  mark  your  answer. 

This  completes  the  experimental  skill  qualification  test  for  M-O-S  51B.  Please 
stop  writing  and  await  further  instructions.  Thank  you. 
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APPENDIX  D 


WRITTEN  TEST 


DIRECTIONS:  Choose  one  NAME 

answer  to  each  item  and  circle  SSAN 

the  appropriate  letter.  PAYGRADE 

d Tool  Maintenance  and  Material  Preparation 

1.  For  jointing,  the  saw  is  placed  in  a vise.  Which  is  the  correct 

position  for  gripping  the  saw? 

a.  Place  the  saw  in  the  vise  with  the  teeth  about  2 inches 

above  the  vise  jaws. 

b.  Place  the  saw  in  the  vise  so  the  gullets  of  the  teeth  are 

about  1/4  inch  above  the  edge  of  the  vise  jaws. 

c.  Place  the  saw  in  the  vise  so  that  the  blade  sits  at  an  angle 

- the  heel  should  be  higher  than  the  toe. 

d.  Place  the  saw  in  the  vise  so  that  the  jaws  of  the  vise  grip 

the  bottom  of  the  saw  blade. 

2.  Which  is  the  correct  method  for  jointing  the  saw  teeth? 

a.  Place  a triangular  file  in  the  jointer  and  move  it  lightly 
over  the  saw  teeth  from  heel  to  toe,  without  rocking  the  file. 

b.  Place  a mill  file  in  the  jointer  and  move  it  lightly  over 
the  saw  teeth  from  heel  to  toe,  without  rocking  the  file. 

c.  Place  a mill  file  in  the  jointer  and  while  applying  pressure, 
move  it  over  the  saw  teeth  from  heel  to  toe,  without  rocking 
the  fUe. 

I d.  Place  a mill  file  in  the  jointer  and  lightly  rock  it  back 

j 

I and  forth  over  the  saw  teeth,  moving  from  heel  to  toe. 


What  is  the  correct  procedure  for  sharpening  a crosscut  handsaw? 

a.  Hold  the  file  at  a right  angle  to  the  saw  blade  and  begin 

at  the  heel  and  work  toward  the  toe. 

b.  Hold  the  file  at  a right  angle  to  the  saw  blade  and  begin 

at  the  midpoint  of  the  saw  and  work  toward  the  heel;  then 
go  back  to  the  midpoint  of  the  saw  and  work  toward  the  toe. 

c.  Hold  the  file  at  a 45°  to  60°  angle  to  the  saw  blade  and 
begin  at  the  heel  and  work  toward  the  toe. 

d.  Hold  the  file  at  a right  angle  to  the  saw  blade  and  begin 

at  the  toe  and  work  toward  the  heel. 

Braces  for  a wall  form  require  2x4's  cut  to  67"  lengths. 

In  order  to  save  materials,  from  which  stack  of  2x4's 

would  you  take  your  lumber? 

a.  12'  - 2x4's. 

b.  8'  - 2x4's. 

From  the  stack  you  selected  in  the  above  question,  how  many  boards 

will  it  take  to  cut  14  - 67"  lengths? 

a.  4 boards. 

b.  7 boards. 

c.  12  boards. 

d.  14  boards. 

How  would  you  measure  and  mark  the  toe  plate  for  studs  which 

are  to  be  placed  12"  on  center? 


a. 

Measure  12  3/4",  draw  a line, 

side  of  the  line. 

and  mark 

an 

"X" 

on 

the 

left 

b. 

Measure  12",  draw  a line,  and 

of  the  line. 

mark  an 

"X" 

on 

the 

left 

side 

c. 

Measure  11",  draw  a line,  and 

of  the  line. 

mark  an 

"X" 

on 

the 

left 

side 

d. 

Measure  12  3/4",  draw  a line, 

side  of  the  line. 

and  mark 

an 

"X" 

on 

the 

right 
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7.  Usins;  the  information  given  in  b'igure  13-1,  what  is  the  eorreet 
spreader  length? 

a . 6" . 

b . «" . 
e.  10". 
d 12". 

8.  Referring  again  to  Figure  13-1,  select  the  correct  spreaders  for 
constructing  the  wall  fom; 

a.  10"  - 1x2  with  holes  drilled  dead  center. 

b.  10"  - 1x2  with  holes  drilled  off  center. 

c.  8"  - 1x2  (squared  off)  with  no  holes  drilled. 

d.  8"  - 1x2  (squared  off)  with  holes  drilled  off  center. 

9.  What  is  the  best  angle  between  the  saw  and  the  work? 

a.  10^^. 

b . 45*^* . 

c . 75‘’ . 

d.  90^^. 

10.  Which  nails  are  best  suited  for  fabricating  concrete  forms? 

a . 8d  common . 

b . I6d  common . 

c.  Finishing  nails. 

d . Double  headed  nails . \ 

11.  To  provide  the  greatest  holding  power,  nails  should  be  driven; 

j 

a.  Straight  and  parallel  to  each  other.  j 

b.  At  an  angle  slightly  toward  each  other.  j 

j 

Erecting  Wall  Forms  j 

t 

12.  If  1x2  spreaders  are  used  and  each  has  a 1/4"  hole  drilled  about  j 

1 1/2"  from  the  end,  they  have  been:  ■ 

a . Drilled  properly . | 

b . Drilled  improperly . 
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13.  The  2x4  wale  is  instaUed  so  that  the  broad  side  faces  the 

studding  and  the  end  extends  about  6"  beyond  the  panels.  The 

wale  has  been: 

a.  Properly  installed. 

b.  Improperly  installed. 

14.  Holes  for  the  tie  wire  are  drilled  so  that  there  is  one  hole  on 

either  side  of  each  stud  requiring  ties.  Both  holes  are  drilled 

above  the  wale.  The  holes  for  the  tie  wires  have  been; 

a . Properly  drilled . 

b.  Improperly  drilled. 

15.  The  tie  wires  are  twisted  from  the  center,  using  16d  nails,  until 

aU  are  of  approximately  uniform  tension.  There  is  a smaU  loop 

left  in  the  center  of  each  tie  wire  and  the  spreaders  resist 

movement.  The  tire  wires  have  been: 

a . Properly  twisted . 

b . Improperly  twisted . 

Placing  and  Finishing  Concrete 

16.  Concrete  wUl  be  poured  into  a 48"  high  form.  At  what  level 

should  you  begin  to  vibrate  the  concrete? 

a.  6". 

b.  18". 

c.  36". 

d.  48". 

17.  Which  is  the  correct  position  for  storing  the  wheelbarrow? 

a.  Lying  upside-down  on  the  ground. 

b.  Lying  on  its  side  on  the  ground. 

c.  Sitting  upright  on  the  ground. 

d.  Tilted  up  against  the  side  of  a building  with  the  wheels 
off  the  ground. 
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18.  Which  is  the  correct  techniiiue? 

a.  Two  men  on  eith»'r  side  of  the  slab  shouUl  lilt  the  screed  at  an 
antjle  and  scrape  the  excess  a^jjre^jate  from  the  slab. 

b.  Two  men  on  either  side  of  the  slab  should  position  the  screed  flat 
on  the  slab,  and  be^innin^  in  the  center  of  the  slab,  use  a sawinjj 
motion . 

c. «  Two  men  on  either  siile  of  the  slab  should  position  the  screed  flat 

on  the  slab,  and  beg;innintc  at  one  end,  should  use  a sawinji  motion. 

d.  Two  men  on  either  side  of  the  slab  should  position  the  screed  flat 
on  the  slab,  and  bet^innin^  at  one  end,  should  use  a sawing  motion. 
They  should  screed  a short  distance,  fill  in  the  depressions,  and 
screed  again  until  there  are  no  depressions. 

19.  Which  is  the  correct  method  for  floating  concrete? 

a.  Hold  the  float  in  a position  so  that  the  tip  can  dig  into  the  con- 
crete. I'se  long  strokes  to  smooth  the  concrete. 

b.  With  the  side  edge  of  the  float,  use  long  strokes  to  smooth  the 
concrete . 

c.  With  the  side  edge  of  the  float,  use  short  brisk  stivkes  to  smiH>th 
the  concrete. 

20.  A slab  is  ivady  to  be  floated  when: 

a.  The  water  sheen  has  disappeared  and  after  stepping  in  and  out  of 
the  mixture  only  a slight  imprint  is  left. 

b.  Water  is  standing  on  top  of  the  mixture  and  after  stepping  in  and 
out  of  the  mixture  no  imprint  is  left. 

21.  When  is  the  concrete  ready  for  first  troweling? 

a . As  swn  as  it  is  set . 

b.  Immediately  after  the  concrete  is  floated. 

c.  After  the  moisture  sheen  has  disappeared  from  the  surface. 

d.  While  the  concrete  is  still  fresh  enough  to  work  the  water  uyi  to  the 
surface . 
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Figure  D-1.  Wall  Form 
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QUESTIONNAIRE 

The  purpose  of  this  questionnaire  is  to  obtain  your  personal  reactions  to  the 
television  and  written  tests.  Check  the  appropriate  categories  and  fill  in  the 
information  at  the  end  of  the  questionnaire.  Please  be  candid  - this 
questionnaire  is  anonymous. 

1.  To  what  extent  was  either  test  a fair  measure  of  your  ability  to  per- 
form in  your  MOS? 


Television  Test 

Written  Test 

Extremely  fair 

Extremely  fair 

Very  fair 

Very  fair 

Somewhat  fair 

Somewhat  fair 

Not  fair 

Not  fair 

How  interesting  was  either  test? 

Television  Test 

Written  Test 

Extremely  interesting 

Extremely  interesting 

Very  interesting 

Very  interesting 

Somewhat  interesting 

Somewhat  interesting 

Not  interesting 

Not  interesting 

How  difficult  was  either  test? 

Television  Test 

Written  Test 

Extremely  difficult 

Extremely  difficult 

Very  difficult 

Very  difficult 

Somewhat  difficult 

Somewhat  difficult 

Not  difficult 

Not  difficult 

Overall,  to  what  extent  were  the  visuals  (pictures,  graphics,  titles. 

etc . ) clear  in  the  television  test? 

Extremely  clear 

Very  clear 

Somewhat  clear 

Not  clear 
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5.  In  the  case  of  the  television  test,  to  what  extent  was  the  narration  easy 
to  understand? 

Extremely  easy  to  understand 

Very  easy  to  understand 

Somewhat  easy  to  understand 

Not  easy  to  understand 

6.  In  the  case  of  the  television  test,  to  what  extent  was  the  answer  sheet 
easy  to  use? 

Extremely  easy  to  use 

Very  easy  to  use 

Somewhat  easy  to  use 

Not  easy  to  use 

7.  Overall,  did  you  have  enough  time  to  answer  the  questions  to  the  tele- 
vision test? 

More  than  enough  time 

Enough  time 

Barely  enough  :ime 

Not  enough  time 

8.  What  is  your  feeling  about  the  overall  pace  (rate  of  presentation)  of  the 
television  test? 

The  pace  was  much  too  slow 

The  pace  somewhat  too  slow 

The  pace  was  somewhat  too  fast 

The  pace  was  much  too  fast 

9.  What  is  your  feeling  about  the  overall  selection  of  items  (situations)  for 
the  television  test? 

The  items  were  extremely  well  chosen 

The  items  were  very  well  chosen 

The  items  were  fairly  well  chosen 

The  items  were  poorly  chosen 

10.  From  where  you  were  sitting,  how  well  were  you  able  to  see  the  tele- 


vision screen?  ^ , 

Extremely  well  (Please  show  your  j Tiy]  1 

Very  well  seating  position  in  1 [ Room 

Fairly  well  the  diagram  on  the  [ ! 

Not  well  right)  1 1 
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Can  you  rtvall  any  sptvi/io  ilemi  (.situations)  in  the  television  test  that 
were  oonfusin^j”'  If  so.  desoribe  the  item(.s)  in  a few  words 


Do  you  have  any  additional  oomments  on  the  television  test"’ 


Can  you  recall  any  specific  items  in  the  written  test  that  were  confusing? 
If  so,  describe  the  itemfs)  in  a few  words. 


Do  you  have  arvy  additional  comments  on  the  written  test? 


